Accessibility techinques for presentation of symbolic expressions

ABSTRACT

Methods for presenting symbolic expressions such as mathematical, scientific, or chemical expressions, formulas, or equations are performed by a computing device. One method includes: displaying a first portion of a symbolic expression within a first area of a display screen; while in a first state in which the first area is selected for aural presentation, aurally presenting first information related to the first portion of the symbolic expression; while in the first state, detecting particular user input; in response to detecting the particular user input, performing the steps of: transitioning from the first state to a second state in which a second area, of the display, is selected for aural presentation; determining second information associated with a second portion, of the symbolic expression, that is displayed within the second area; in response to determining the second information, aurally presenting the second information.

TECHNICAL FIELD

The disclosed embodiments relate generally to computer-implementedtechniques for presenting information, and more particularly, tocomputer-implemented techniques for presenting symbolic expressions suchas mathematical, scientific, or chemical expressions, formulas, orequations in a more accessible way.

BACKGROUND

Computers are very powerful tools for presenting information to users.Often, computers present information visually, on a display screen orother visual presentation device. However, many users of computers areblind, visually impaired, dyslexic or have low-vision or learningdisabilities, or are sighted users who simply want or need to use acomputing device without looking at the device during operation. Theseusers can benefit from accessibility devices that present information innonvisual ways. One way a computer presents information a nonvisual wayis with a text-to-speech transducer that reads aloud the content of adisplay screen.

Current text-to-speech systems work reasonably well with linear prose,like that found in an electronic message or novel. Unfortunately,existing text-to-speech systems remain cumbersome and difficult to usefor presenting symbolic expressions such as mathematical, scientific, orchemical expressions, formulas, or equations. Precise navigation andpresentation of symbolic expressions is often difficult or not possible,thereby creating a significant cognitive burden on a user with impairedvision.

Accordingly, there is a need for computer-implemented techniques forpresenting symbolic expressions in a more accessible way. Suchtechniques may complement or replace existing computer-implementedtechniques for presenting information. Such techniques reduce thecognitive burden on a user with impaired vision and produce a moreaccessible and efficient human-machine interface.

SUMMARY

The above deficiencies and other accessibility problems associated withexisting text-to-speech systems for presenting symbolic expressions arereduced or eliminated by the techniques disclosed herein. The techniquesare implemented by a computing device. In some embodiments, thecomputing device is a desktop computer. In some embodiments, thecomputing device is a portable computer such as a laptop computer orhandheld device. In some embodiments, the computing device has atouchpad (also known as a “track pad”). In some embodiments, thecomputing device has a touch-sensitive display (also known as a “touchscreen” or “touch screen display”). In some embodiments, the computingdevice has a graphical user interface (GUI), one or more processors,memory and one or more modules, programs or sets of instructions storedin the memory for presenting symbolic expressions. In some embodiments,the symbolic expressions may include mathematical, scientific, orchemical formulas, equations, or expressions.

In some embodiments, a method for presenting a symbolic expression isperformed at a computing device with a display. The method includesdisplaying a first portion of the symbolic expression within a firstarea of the display. The method also including aurally presenting firstinformation related to the first portion of the symbolic expressionwhile the computing device is in a first state in which the first areais selected for aural presentation. Also while the computing is in thefirst state, the computing device detects particular user input and inresponse to detecting the particular user input, the computing devicetransitions from the first state to a second state in which a secondarea of the display is selected for aural presentation. In the secondstate, second information associated with a second portion of thesymbolic expression that is displayed within the second area of thedisplay is determined and aurally presented.

In some embodiments, the computing device has a touch-sensitive surfaceand detecting the particular user input includes detecting a userinterface navigation gesture on the touch-sensitive surface. In someembodiments, the display is a touch-screen display and thetouch-sensitive surface is on the display. In some embodiments, the userinterface navigation gesture is a multi-finger gesture. In someembodiments, the user interface navigation gesture is independent ofcontacting a location on the touch-sensitive surface that corresponds tothe second area. In some embodiments, the user interface navigationgesture does not contact a location on the touch-sensitive surface thatcorresponds to the second area. In some embodiments, the user interfacenavigation gesture is dependent on contacting a location on thetouch-sensitive surface that corresponds to the second area.

In some embodiments, the technique also includes, while in the firststate, mapping the first area of the display to the touch-sensitivesurface. In response to detecting the particular user input, thecomputing devices ceases to map the first area of the display to thetouch-sensitive surface, and proportionally maps the second area of thedisplay to be substantially coextensive with the touch-sensitivesurface.

In some embodiments, the symbolic expression is a mathematical,scientific, or chemical expression, equation, or formula.

In some embodiments, detecting the particular user input includesdetecting a pointing device action. In some embodiments, the pointingdevice is a mouse.

In some embodiments, detecting the particular user input includesdetecting a key strike on a keyboard.

In some embodiments, the second area resides within the first area andthe second portion of the symbolic expression is a part of the firstportion.

In some embodiments, the first information includes one or more words,aurally presented as spoken text, describing the first portion of thesymbolic expression. In some embodiments, the second informationincludes one or more words, aurally presented as spoken text, describingthe second portion of the symbolic expression.

In some embodiments, the computing device, in response to determiningthe second information, outputs a transition sound to indicate that thearea selected for aural presentation has transitioned from the firstarea to the second area.

In some embodiments, the first area resides within the second area andthe first portion of the symbolic expression is a part of the secondportion.

In some embodiments, a computing device with a touch-sensitive surface,a display, one or more processors, memory is configured by one or moreprograms stored in the memory to perform some of the above methods.

In some embodiments, one or more non-transitory computer-readable mediastore one or more programs which, when executed by a computing devicewith a display, cause the computing device to perform some of the abovemethods.

In some embodiments, a graphical user interface on a computing devicewith a display comprises a symbolic expression displayed on the displayand a plurality of selectable areas of the display. The symbolicexpression has a plurality of portions. Each selectable area of theplurality of selectable areas corresponds to a portion of the pluralityof portions. While in a first state in which a first selectable area ofthe plurality of selectable areas is selected for aural presentation,first information associated with the portion of the symbolic expressioncorresponding to the first selectable area is aurally presented. Whilein the first state, user input selecting a second selectable area of theplurality of selectable areas is detected. In response to detecting theuser input, second information associated with the portion of thesymbolic expression corresponding to the second selectable area isaurally presented.

In some embodiments, a method for presenting a symbolic expressionincludes while in a first state in which a first portion of a symbolicexpression is selected for presentation, presenting first informationrelated to the first portion of the symbolic expression. While in thefirst state, detecting particular user input. In response to detectingthe particular user input: transitioning from the first state to asecond state in which a second portion of the symbolic expression isselected for presentation; determining second information associatedwith a second portion of the symbolic expression; and in response todetermining the second information, presenting the second information.In some embodiments, the computing device has or is operatively coupledto an electro-mechanical braille display and presenting the secondinformation includes sending the information to the electro-mechanicalbraille display.

Thus, computing devices with displays are provided with new and improvedaccessibility methods and user interfaces for navigating symbolicexpressions, thereby increasing the effectiveness, efficiency, and usersatisfaction with such devices. Such methods, devices, and interfacesmay complement or replace existing accessibility methods, devices, andinterfaces.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the aforementioned embodiments of theinvention as well as additional embodiments thereof, reference should bemade to the Description of Embodiments below, in conjunction with thefollowing drawings in which like reference numerals refer tocorresponding parts throughout the figures.

FIG. 1 is a block diagram of a portable computing device with atouch-sensitive display, according to some embodiments of the invention.

FIG. 2 illustrates a portable computing device in accordance with someembodiments.

FIG. 3 is a block diagram of an exemplary computing device with adisplay and a touch-sensitive surface in accordance with someembodiments.

FIGS. 4A-4P illustrate accessibility user interfaces for a portablecomputing device with a touch-sensitive surface in accordance with someembodiments.

FIGS. 5A-5F illustrate tree representations of symbolic expressions inaccordance with some embodiments.

FIGS. 6A-6F illustrate accessibility user interfaces that map respectiveselectable areas corresponding to portions of a symbolic expression to atouch-sensitive surface in accordance with some embodiments.

FIG. 7 is a flow diagram illustrating an accessibility method fornavigation among portions of a symbolic expression in accordance withsome embodiments.

DESCRIPTION OF EMBODIMENTS Introduction

The following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, these elements should notbe limited by these terms. These terms are only used to distinguish oneelement from another. For example, a first portion could be termed asecond portion, and, similarly, a second portion could be termed a firstportion, without departing from the scope of the present invention. Thefirst portion and the second portion are both portions, but they are notthe same portion, unless otherwise clearly indicated.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an”, and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. Use of “criteria” may indicate either or both of the singularusage of the term, “criterion”, or the plural form “criteria”, or viceversa.

It will also be understood that the term “and/or” as used herein refersto and encompasses any and all possible combinations of one or more ofthe associated listed items. It will be further understood that theterms “comprises” and/or “comprising”, when used herein, specify thepresence of stated features, integers, steps, operations, elements,components and/or groups, but do not preclude the presence or additionof one or more other features, integers, steps, operations, elements,components, and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in response to detecting,” dependingon the context. Similarly, the phrase “if it is determined” or “if [astated condition or event] is detected” may be construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event” or “in response to detecting [the statedcondition or event],” depending on the context.

Exemplary Computing Devices and User Interfaces

Embodiments of computing devices, user interfaces for such devices, andassociated processes for using such devices are described. In someembodiments, the computing device is a portable computing device such asa tablet computer or electronic reading device that may also containother functions, such as PDA and/or music player functions. Exemplaryembodiments of portable computing devices include, without limitation,the iPad® device from Apple, Inc. of Cupertino, Calif. In someembodiments, the computing device is a stationary computing device suchas a desktop computer or workstation computer.

In the discussion that follows, a computing device that includes adisplay and a touch-sensitive surface through which user input isprovided is described. It should be understood, however, that thecomputing device may include one or more other physical user interfacedevices, such as a physical keyboard, a mouse and/or a joystick throughwhich user input is provided in addition to or instead of user inputprovided through a touch-sensitive surface.

In some embodiments, the device supports one or more applications thatimplement or use the techniques described herein for presentation ofsymbolic expressions. These applications may include, but are notlimited to, an e-book application, an e-reader application, apresentation application, a word processing application, a web browsingapplication, a spreadsheet application, or an e-mail application.

The various applications that may be executed on the device may use atleast one common physical user interface device, such as thetouch-sensitive surface. One or more functions of the touch-sensitivesurface as well as corresponding information displayed on the device maybe adjusted and/or varied from one application to the next and/or withina respective application. In this way, a common physical architecture(such as the touch-sensitive surface) of the device may support thevariety of applications with user interfaces that are intuitive andtransparent.

The user interfaces may include one or more soft keyboard embodiments.The soft keyboard embodiments may include standard (QWERTY) and/ornon-standard configurations of symbols on the displayed icons of thekeyboard, such as those described in U.S. patent application Ser. No.11/459,606, “Keyboards For Portable Electronic Devices,” filed Jul. 24,2006, and Ser. No. 11/459,615, “Touch Screen Keyboards For PortableElectronic Devices,” filed Jul. 24, 2006, the contents of which arehereby incorporated by reference in their entirety. The keyboardembodiments may include a reduced number of icons (or soft keys)relative to the number of keys in existing physical keyboards, such asthat for a typewriter. This may make it easier for users to select oneor more icons in the keyboard, and thus, one or more correspondingsymbols. The keyboard embodiments may be adaptive. For example,displayed icons may be modified in accordance with user actions, such asselecting one or more icons and/or one or more corresponding symbols.One or more applications on the device may utilize common and/ordifferent keyboard embodiments. Thus, the keyboard embodiment used maybe tailored to at least some of the applications. In some embodiments,one or more keyboard embodiments may be tailored to a respective user.For example, one or more keyboard embodiments may be tailored to arespective user based on a word usage history (lexicography, slang,individual usage) of the respective user. Some of the keyboardembodiments may be adjusted to reduce a probability of a user error whenselecting one or more icons, and thus one or more symbols, when usingthe soft keyboard embodiments.

Attention is now directed towards an embodiment of a portable computingdevice with a touch-sensitive display. FIG. 1 is a block diagramillustrating a portable computing device 100 with a touch-sensitivedisplay 112 in accordance with some embodiments. The touch-sensitivedisplay 112 is sometimes called a “touch screen” for convenience, andmay also be known as or called a touch-sensitive display system. Thedevice 100 may include a memory 102 (which may include one or morenon-transitory computer readable mediums), a memory controller 122, oneor more processing units (CPU's) 120, a peripherals interface 118, RFcircuitry 108, audio circuitry 110, a speaker 111, a microphone 113, aninput/output (I/O) subsystem 106, other input or control devices 116,and an external port 124. The device 100 may include one or more opticalsensors 164. These components may communicate over one or morecommunication buses or signal lines 103.

It should be appreciated that the device 100 is only one example of aportable computing device 100, and that the device 100 may have more orfewer components than shown, may combine two or more components, or amay have a different configuration or arrangement of the components. Thevarious components shown in FIG. 1 may be implemented in hardware,software, or a combination of both hardware and software, including oneor more signal processing and/or application specific integratedcircuits.

Memory 102 may include high-speed random access memory and may alsoinclude non-volatile memory, such as one or more magnetic disk storagedevices, flash memory devices, or other non-volatile solid-state memorydevices. Access to memory 102 by other components of the device 100,such as the CPU 120 and the peripherals interface 118, may be controlledby the memory controller 122.

The peripherals interface 118 couples the input and output peripheralsof the device to the CPU 120 and memory 102. The one or more processors120 run or execute various software programs and/or sets of instructionsstored in memory 102 to perform various functions for the device 100 andto process data.

In some embodiments, the peripherals interface 118, the CPU 120, and thememory controller 122 may be implemented on a single chip, such as achip 104. In some other embodiments, they may be implemented on separatechips.

The RF (radio frequency) circuitry 108 receives and sends RF signals,also called electromagnetic signals. The RF circuitry 108 convertselectrical signals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. The RF circuitry 108 may include well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. The RFcircuitry 108 may communicate with networks, such as the Internet, alsoreferred to as the World Wide Web (WWW), an intranet and/or a wirelessnetwork, such as a cellular telephone network, a wireless local areanetwork (LAN) and/or a metropolitan area network (MAN), and otherdevices by wireless communication. The wireless communication may useany of a plurality of communications standards, protocols andtechnologies, including but not limited to Global System for MobileCommunications (GSM), Enhanced Data GSM Environment (EDGE), high-speeddownlink packet access (HSDPA), wideband code division multiple access(W-CDMA), code division multiple access (CDMA), time division multipleaccess (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a,IEEE 802.11b, IEEE 802.11g and/or IEEE 802.11n), voice over InternetProtocol (VoIP), Wi-MAX, a protocol for email (e.g., Internet messageaccess protocol (IMAP) and/or post office protocol (POP)), instantmessaging (e.g., extensible messaging and presence protocol (XMPP),Session Initiation Protocol for Instant Messaging and PresenceLeveraging Extensions (SIMPLE), Instant Messaging and Presence Service(IMPS)), and/or Short Message Service (SMS)), or any other suitablecommunication protocol, including communication protocols not yetdeveloped as of the filing date of this document.

The audio circuitry 110, the speaker 111, and the microphone 113 providean audio interface between a user and the device 100. The audiocircuitry 110 receives audio data from the peripherals interface 118,converts the audio data to an electrical signal, and transmits theelectrical signal to the speaker 111. The speaker 111 converts theelectrical signal to human-audible sound waves. The audio circuitry 110also receives electrical signals converted by the microphone 113 fromsound waves. The audio circuitry 110 converts the electrical signal toaudio data and transmits the audio data to the peripherals interface 118for processing. Audio data may be retrieved from and/or transmitted tomemory 102 and/or the RF circuitry 108 by the peripherals interface 118.In some embodiments, the audio circuitry 110 also includes a headsetjack (e.g. 212, FIG. 2). The headset jack provides an interface betweenthe audio circuitry 110 and removable audio input/output peripherals,such as output-only headphones or a headset with both output (e.g., aheadphone for one or both ears) and input (e.g., a microphone).

The I/O subsystem 106 couples input/output peripherals on the device100, such as the touch screen 112 and other input/control devices 116,to the peripherals interface 118. The I/O subsystem 106 may include adisplay controller 156 and one or more input controllers 160 for otherinput or control devices. The one or more input controllers 160receive/send electrical signals from/to other input or control devices116. The other input/control devices 116 may include physical buttons(e.g., push buttons, rocker buttons, etc.), dials, slider switches,joysticks, click wheels, and so forth. In some alternate embodiments,input controller(s) 160 may be coupled to any (or none) of thefollowing: a keyboard, infrared port, USB port, and a pointer devicesuch as a mouse. The one or more buttons (e.g., 208, FIG. 2) may includean up/down button for volume control of the speaker 111 and/or themicrophone 113. The one or more buttons may include a push button (e.g.,206, FIG. 2). A quick press of the push button may disengage a lock ofthe touch screen 112 or begin a process that uses gestures on the touchscreen to unlock the device, as described in U.S. patent applicationSer. No. 11/322,549, “Unlocking a Device by Performing Gestures on anUnlock Image,” filed Dec. 23, 2005, which is hereby incorporated byreference in its entirety. A longer press of the push button (e.g., 206)may turn power to the device 100 on or off. The user may be able tocustomize a functionality of one or more of the buttons. The touchscreen 112 is used to implement virtual or soft buttons and one or moresoft keyboards.

The touch-sensitive touch screen 112 provides an input interface and anoutput interface between the device and a user. The display controller156 receives and/or sends electrical signals from/to the touch screen112. The touch screen 112 displays visual output to the user. The visualoutput may include graphics, text, icons, video, and any combinationthereof (collectively termed “graphics”). In some embodiments, some orall of the visual output may correspond to user interface objects.

A touch screen 112 has a touch-sensitive surface, sensor or set ofsensors that accepts input from the user based on haptic and/or tactilecontact. The touch screen 112 and the display controller 156 (along withany associated modules and/or sets of instructions in memory 102) detectcontact (and any movement or breaking of the contact) on the touchscreen 112 and converts the detected contact into interaction with userinterface objects (e.g., one or more soft keys, icons, web pages orimages) that are displayed on the touch screen. In an exemplaryembodiment, a point of contact between a touch screen 112 and the usercorresponds to a finger of the user.

The touch screen 112 may use LCD (liquid crystal display) technology, orLPD (light emitting polymer display) technology, although other displaytechnologies may be used in other embodiments. The touch screen 112 andthe display controller 156 may detect contact and any movement orbreaking thereof using any of a plurality of touch sensing technologiesnow known or later developed, including but not limited to capacitive,resistive, infrared, and surface acoustic wave technologies, as well asother proximity sensor arrays or other elements for determining one ormore points of contact with a touch screen 112. In an exemplaryembodiment, projected mutual capacitance sensing technology is used,such as that found in the iPad® from Apple Computer, Inc. of Cupertino,Calif.

A touch-sensitive display in some embodiments of the touch screen 112may be analogous to the multi-touch sensitive tablets described in thefollowing U.S. Pat. No. 6,323,846 (Westerman et al.), U.S. Pat. No.6,570,557 (Westerman et al.), and/or U.S. Pat. No. 6,677,932(Westerman), and/or U.S. Patent Publication 2002/0015024A1, each ofwhich is hereby incorporated by reference in its entirety. However, atouch screen 112 displays visual output from the portable device 100,whereas touch sensitive tablets do not provide visual output.

A touch-sensitive display in some embodiments of the touch screen 112may be as described in the following applications: (1) U.S. patentapplication Ser. No. 11/381,313, “Multipoint Touch Surface Controller,”filed May 2, 2006; (2) U.S. patent application Ser. No. 10/840,862,“Multipoint Touchscreen,” filed May 6, 2004; (3) U.S. patent applicationSer. No. 10/903,964, “Gestures For Touch Sensitive Input Devices,” filedJul. 30, 2004; (4) U.S. patent application Ser. No. 11/048,264,“Gestures For Touch Sensitive Input Devices,” filed Jan. 31, 2005; (5)U.S. patent application Ser. No. 11/038,590, “Mode-Based Graphical UserInterfaces For Touch Sensitive Input Devices,” filed Jan. 18, 2005; (6)U.S. patent application Ser. No. 11/228,758, “Virtual Input DevicePlacement On A Touch Screen User Interface,” filed Sep. 16, 2005; (7)U.S. patent application Ser. No. 11/228,700, “Operation Of A ComputerWith A Touch Screen Interface,” filed Sep. 16, 2005; (8) U.S. patentapplication Ser. No. 11/228,737, “Activating Virtual Keys Of ATouch-Screen Virtual Keyboard,” filed Sep. 16, 2005; and (9) U.S. patentapplication Ser. No. 11/367,749, “Multi-Functional Hand-Held Device,”filed Mar. 3, 2006. All of these applications are incorporated byreference herein in their entirety.

The touch screen 112 may have a resolution in excess of 100 dpi. In anexemplary embodiment, the touch screen has a resolution of approximately160 dpi. The user may make contact with the touch screen 112 using anysuitable object or appendage, such as a stylus, a finger, and so forth.In some embodiments, the user interface is designed to work primarilywith finger-based contacts and gestures, which are much less precisethan stylus-based input due to the larger area of contact of a finger onthe touch screen. In some embodiments, the device translates the roughfinger-based input into a precise pointer/cursor position or command forperforming the actions desired by the user.

In some embodiments, in addition to the touch screen, the device 100 mayinclude a touchpad (not shown) for activating or deactivating particularfunctions. In some embodiments, the touchpad is a touch-sensitive areaof the device that, unlike the touch screen, does not display visualoutput. The touchpad may be a touch-sensitive surface that is separatefrom the touch screen 112 or an extension of the touch-sensitive surfaceformed by the touch screen.

In some embodiments, the device 100 may include a physical or virtualclick wheel as an input control device 116. A user may navigate amongand interact with one or more graphical objects (e.g., icons) displayedin the touch screen 112 by rotating the click wheel or by moving a pointof contact with the click wheel (e.g., where the amount of movement ofthe point of contact is measured by its angular displacement withrespect to a center point of the click wheel). The click wheel may alsobe used to select one or more of the displayed icons. For example, theuser may press down on at least a portion of the click wheel or anassociated button. User commands and navigation commands provided by theuser via the click wheel may be processed by an input controller 160 aswell as one or more of the modules and/or sets of instructions in memory102. For a virtual click wheel, the click wheel and click wheelcontroller may be part of the touch screen 112 and the displaycontroller 156, respectively. For a virtual click wheel, the click wheelmay be either an opaque or semitransparent object that appears anddisappears on the touch screen display in response to user interactionwith the device. In some embodiments, a virtual click wheel is displayedon the touch screen of a portable computing device and operated by usercontact with the touch screen.

The device 100 also includes a power system 162 for powering the variouscomponents. The power system 162 may include a power management system,one or more power sources (e.g., battery, alternating current (AC)), arecharging system, a power failure detection circuit, a power converteror inverter, a power status indicator (e.g., a light-emitting diode(LED)) and any other components associated with the generation,management and distribution of power in portable devices.

The device 100 may also include one or more optical sensors 164. FIG. 1shows an optical sensor coupled to an optical sensor controller 158 inI/O subsystem 106. The optical sensor 164 may include charge-coupleddevice (CCD) or complementary metal-oxide semiconductor (CMOS)phototransistors. The optical sensor 164 receives light from theenvironment, projected through one or more lens, and converts the lightto data representing an image. In conjunction with an imaging modulesoftware component (also called a camera module) stored in the memory102, the optical sensor 164 may capture still images or video. In someembodiments, an optical sensor is located on the back of the device 100,opposite the touch screen display 112 on the front of the device, sothat the touch screen display may be used as a viewfinder for stilland/or video image acquisition. In some embodiments, an optical sensoris located on the front of the device so that the user's image may beobtained for videoconferencing while the user views the other videoconference participants on the touch screen display. In someembodiments, the position of the optical sensor 164 can be changed bythe user (e.g., by rotating the lens and the sensor in the devicehousing) so that a single optical sensor 164 may be used along with thetouch screen display for both video conferencing and still and/or videoimage acquisition.

The device 100 may also include one or more proximity sensors 166. FIG.1 shows a proximity sensor 166 coupled to the peripherals interface 118.Alternately, the proximity sensor 166 may be coupled to an inputcontroller 160 in the I/O subsystem 106. The proximity sensor 166 mayperform as described in U.S. patent application Ser. No. 11/241,839,“Proximity Detector In Handheld Device”; Ser. No. 11/240,788, “ProximityDetector In Handheld Device”; Ser. No. 11/620,702, “Using Ambient LightSensor To Augment Proximity Sensor Output”; Ser. No. 11/586,862,“Automated Response To And Sensing Of User Activity In PortableDevices”; and Ser. No. 11/638,251, “Methods And Systems For AutomaticConfiguration Of Peripherals,” which are hereby incorporated byreference in their entirety. In some embodiments, the proximity sensorturns off and disables the touch screen 112 when the computing device isplaced near the user's ear (e.g., when the user is making a phone call).In some embodiments, the proximity sensor keeps the screen off when thedevice is in the user's pocket, purse, or other dark area to preventunnecessary battery drainage when the device is a locked state.

The device 100 may also include one or more accelerometers 168. FIG. 1shows an accelerometer 168 coupled to the peripherals interface 118.Alternately, the accelerometer 168 may be coupled to an input controller160 in the I/O subsystem 106. The accelerometer 168 may perform asdescribed in U.S. Patent Publication No. 20050190059,“Acceleration-based Theft Detection System for Portable ElectronicDevices,” and U.S. Patent Publication No. 20060017692, “Methods AndApparatuses For Operating A Portable Device Based On An Accelerometer,”both of which are which are incorporated by reference herein in theirentirety. In some embodiments, information is displayed on the touchscreen display in a portrait view or a landscape view based on ananalysis of data received from the one or more accelerometers.

In some embodiments, the software components stored in memory 102 mayinclude an operating system 126, a communication module (or set ofinstructions) 127, an accessibility module 128, a contact/motion module(or set of instructions) 129, a graphics module (or set of instructions)130, an attachment editing module 131, a text input module (or set ofinstructions) 132, a Global Positioning System (GPS) module (or set ofinstructions) 133, and applications (or sets of instructions) 134.

The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X,WINDOWS, or an embedded operating system such as VxWorks) includesvarious software components and/or drivers for controlling and managinggeneral system tasks (e.g., memory management, storage device control,power management, etc.) and facilitates communication between varioushardware and software components.

The communication module 127 facilitates communication with otherdevices over one or more external ports 124 and also includes varioussoftware components for handling data received by the RF circuitry 108and/or the external port 124. The external port 124 (e.g., UniversalSerial Bus (USB), FIREWIRE, etc.) is adapted for coupling directly toother devices or indirectly over a network (e.g., the Internet, wirelessLAN, etc.). In some embodiments, the external port is a multi-pin (e.g.,30-pin) connector that is the same as, or similar to and/or compatiblewith the 30-pin connector used on some iPad® devices.

In conjunction with audio circuitry 110, speaker 111, touch screen 112,display controller 156, contact module 129, graphics module 130, andtext input module 132, the accessibility module 128 facilitatestouch-based navigation among user interface elements so that a user maynavigate, select, activate, and otherwise interact with elements in theuser interface without necessarily seeing the user interface. In someembodiments, the accessibility module 128 facilitates selecting andactivating user interface elements within the user interface withoutdirectly selecting or contacting those user interface elements.Exemplary user interface elements include, without limitation, userinterface icons and widgets, application icons, application interfaces,menus, web browsers, web pages and applications from the world-wide web,application controls, documents, soft/virtual keyboards and numericpads, calculators, calendars, lists, tables, emails, HTML text, XMLtext, rich text, unformatted text, maps, game interfaces, etc. Userinterface elements include any aspect of a graphical or textual userinterface that a user may interact with or manipulate when using anelectronic device the user interface is running on.

The contact/motion module 129 may detect contact with the touch screen112 (in conjunction with the display controller 156) and other touchsensitive devices (e.g., a touchpad or physical click wheel). Thecontact/motion module 129 includes various software components forperforming various operations related to detection of contact, such asdetermining if contact has occurred (e.g., detecting a finger-downevent), determining if there is movement of the contact and tracking themovement across the touch-sensitive surface (e.g., detecting one or morefinger-dragging events), and determining if the contact has ceased(e.g., detecting a finger-up event or a break in contact). Thecontact/motion module 130 receives contact data from the touch-sensitivesurface. Determining movement of the point of contact, which isrepresented by a series of contact data, may include determining speed(magnitude), velocity (magnitude and direction), and/or an acceleration(a change in magnitude and/or direction) of the point of contact. Theseoperations may be applied to single contacts (e.g., one finger contacts)or to multiple simultaneous contacts (e.g., “multitouch”/multiple fingercontacts). In some embodiments, the contact/motion module 129 and thedisplay controller 156 detects contact on a touchpad. In someembodiments, the contact/motion module 129 and the controller 160detects contact on a click wheel.

The contact/motion module 129 may detect a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns. Thus, a gesture may be detected by detecting a particularcontact pattern. For example, detecting a finger tap gesture comprisesdetecting a finger-down event followed by detecting a finger-up event atthe same position (or substantially the same position) as thefinger-down event (e.g., at the position of an icon). As anotherexample, detecting a finger swipe gesture on the touch-sensitive surfacecomprises detecting a finger-down event followed by detecting one ormore finger-dragging events, and subsequently followed by detecting afinger-up event.

The graphics module 130 includes various known software components forrendering and displaying graphics on the touch screen 112 or otherdisplay, including components for changing the intensity of graphicsthat are displayed. As used herein, the term “graphics” includes anyobject that can be displayed to a user, including without limitationtext, web pages, icons (such as user interface objects including softkeys), digital images, videos, animations and the like.

In some embodiments, the graphics module 130 stores data representinggraphics to be used. Each graphic may be assigned a corresponding code.The graphics module 130 receives, from applications etc., one or morecodes specifying graphics to be displayed along with, if necessary,coordinate data and other graphic property data, and then generatesscreen image data to output to display controller 156.

The text input module 132, which may be a component of graphics module130, provides soft keyboards for entering text in various softwareapplications stored in memory 102 (e.g., contacts, e-mail, IM, browser,and any other application that needs text input).

The GPS module 133 determines the location of the device and providesthis information for use in various software applications stored inmemory 102 (e.g., to telephone application for use in location-baseddialing, to camera application as picture/video metadata, and toapplications that provide location-based services such as weatherwidgets, local yellow page widgets, and map/navigation widgets).

The applications 134 may include the following modules (or sets ofinstructions), or a subset or superset thereof:

an e-mail client module;

a web browser module;

a word processing module;

a spreadsheet module; and

an e-reader module for reading digital books, textbooks, articles,periodicals, or the like.

Examples of other applications 134 that may be stored in memory 102include other word processing applications, other image editingapplications, drawing applications, presentation applications,JAVA-enabled applications, encryption, digital rights management, voicerecognition, and voice replication.

One or more of the above identified modules and applications maycorrespond to a set of executable instructions for performing one ormore functions described above and/or the methods described in thisapplication (e.g., the computer-implemented methods and otherinformation processing methods described herein). These modules (i.e.,sets of instructions) need not be implemented as separate softwareprograms, procedures or modules, and thus various subsets of thesemodules may be combined or otherwise re-arranged in various embodiments.For example, an e-reader module may be combined with a web browsermodule into a single module). In some embodiments, memory 102 may storea subset of the modules and data structures identified above.Furthermore, memory 102 may store additional modules and data structuresnot described above.

In some embodiments, the device 100 is a device where operation of apredefined set of functions on the device is performed exclusivelythrough a touch screen 112 and/or a touchpad. By using a touch screenand/or a touchpad as the primary input/control device for operation ofthe device 100, the number of physical input/control devices (such aspush buttons, dials, and the like) on the device 100 may be reduced.

The predefined set of functions that may be performed exclusivelythrough a touch screen and/or a touchpad include navigation between userinterfaces. In some embodiments, the touchpad, when touched by the user,navigates the device 100 to a main, home, or root menu from any userinterface that may be displayed on the device 100. In such embodiments,the touchpad may be referred to as a “menu button.” In some otherembodiments, the menu button may be a physical push button or otherphysical input/control device instead of a touchpad.

FIG. 2 illustrates a portable computing device 100 having a touch screen112 in accordance with some embodiments. The touch screen may displayone or more graphics within user interface (UI) 200. In this embodiment,as well as others described below, a user may select one or more of thegraphics by making contact or touching the graphics, for example, withone or more fingers 202 (not drawn to scale in the figure). In someembodiments, selection of one or more graphics occurs when the userbreaks contact with the one or more graphics. In some embodiments, thecontact may include a gesture, such as one or more taps, one or moreswipes (from left to right, right to left, upward and/or downward)and/or a rolling of a finger (from right to left, left to right, upwardand/or downward) that has made contact with the device 100. In someembodiments, inadvertent contact with a graphic may not select thegraphic. For example, a swipe gesture that sweeps over an applicationicon may not select the corresponding application when the gesturecorresponding to selection is a tap.

The device 100 may also include one or more physical buttons, such as“home” or menu button 204. As described previously, the menu button 204may be used to navigate to any application 134 in a set of applicationsthat may be executed on the device 100. Alternatively, in someembodiments, the menu button is implemented as a soft key in a GUI intouch screen 112.

In one embodiment, the device 100 includes a touch screen 112, a menubutton 204, a push button 206 for powering the device on/off and lockingthe device, volume adjustment button(s) 208, a head set jack 212, and adocking/charging external port 124. The push button 206 may be used toturn the power on/off on the device by depressing the button and holdingthe button in the depressed state for a predefined time interval; tolock the device by depressing the button and releasing the button beforethe predefined time interval has elapsed; and/or to unlock the device orinitiate an unlock process. In an alternative embodiment, the device 100also may accept verbal input for activation or deactivation of somefunctions through the microphone 113.

FIG. 3 is a block diagram of an exemplary computing device with adisplay and a touch-sensitive surface in accordance with someembodiments. Device 300 need not be portable. In some embodiments, thedevice 300 is a laptop computer, a desktop computer, a tablet computer,a multimedia player device, a navigation device, an educational device(such as a child's learning toy), a gaming system, or a control device(e.g., a home or industrial controller). The device 300 typicallyincludes one or more processing units (CPU's) 310, one or more networkor other communications interfaces 370, memory 380, and one or morecommunication buses 320 for interconnecting these components. Thecommunication buses 320 may include circuitry (sometimes called achipset) that interconnects and controls communications between systemcomponents. The device 300 includes an input/output (I/O) interface 330comprising a display 340, which in some embodiments is a touch screendisplay 112. The I/O interface 330 also may include a keyboard and/ormouse (or other pointing device) 350 and a touchpad 360. Memory 380includes high-speed random access memory, such as DRAM, SRAM, DDR RAM orother random access solid state memory devices; and may includenon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 380 may optionallyinclude one or more storage devices remotely located from the CPU(s)310. In some embodiments, memory 380 stores programs, modules, and datastructures analogous to the programs, modules, and data structuresstored in the memory 102 of portable computing device 100 (FIG. 1), or asubset thereof. Furthermore, memory 380 may store additional programs,modules, and data structures not present in the memory 102 of portablecomputing device 100. For example, memory 380 of device 300 may store adrawing module, a presentation module, a word processing module, awebsite creation module, a disk authoring module, a spreadsheet moduleand/or attachment an editing module, while memory 102 of portablecomputing device 100 (FIG. 1) may not store these modules.

Each of the above identified modules may be stored in one or more of thepreviously mentioned memory devices. Each of the above identifiedmodules may correspond to a set of instructions for performing afunction described above and/or the methods described in thisapplication (e.g., the computer-implemented methods and otherinformation processing methods described herein). The above identifiedmodules or programs (i.e., sets of instructions) need not be implementedas separate software programs, procedures or modules, and thus varioussubsets of these modules may be combined or otherwise re-arranged invarious embodiments. In some embodiments, memory 380 may store a subsetof the modules and data structures identified above. Furthermore, memory370 may store additional modules and data structures not describedabove.

Attention is now directed towards exemplary embodiments of userinterfaces (“UI”) and associated processes that may be implemented on acomputing device with a display and a touch-sensitive surface, such asdevice 300 or portable computing device 100.

Navigation of Symbolic Expressions

The exemplary user interfaces depicted in FIGS. 4A-4P are foraccessibility interfaces for touch-based navigation among presentationelements (i.e., portions) of symbolic expressions on an electronicdevice.

In some embodiments, the symbolic expressions include mathematical,scientific, and chemical formulas, equations, and expressions.

In some embodiments, the presentation elements include tokens elementsand layout elements of symbolic expressions. The tokens elementsrepresent individual symbols of symbolic expressions. The layoutelements represent information for constructing a symbolic expressionfrom one or more symbolic sub-expressions. The set of possible token andlayout elements may vary depending on the domain of the symbolicexpression. For example, the token elements for the mathematics domainmay include identifiers (e.g., variable or function names), numbers,operators (including fences, such as parentheses, and separators, suchas commas), text, string literals, and whitespace. For example, in amathematical equation, formula, or expression, sin and 24 may representa single identifier token and a single number token, respectively. Thelayout elements in the mathematics domain may include a horizontalgrouping of symbolic sub-expressions, a fraction, a square root, aradical, fenced content (e.g., content surround by parentheses),enclosed content (e.g., content enclosed with a stretching symbol suchas a long division sign), subscripts, superscripts,subscript-superscript pairs, underscripts, overscripts,underscript-overscript pairs, prescripts and tensor indices, tables andmatrices, etc. Other domains (e.g., chemical) may include some, all, ornone of these layout elements in addition to other domain-specificlayout elements.

In some embodiments, user interfaces are for audible navigation ofpresentation elements of mathematical formulas, equations, orexpressions. In other embodiments, the user interfaces are for audiblenavigation of presentation elements of other types of symbolicexpressions, such as chemical and scientific formulas, equations, andexpressions. The symbolic expression could also be a description of animage or chart. For example, an image of a graph in a math textbookcould have an accessibility description that is not show onscreen thatsays “Graph of x/(ŷ2)”.

Further, although in some embodiments navigation of presentationelements is accomplished with gestures on a touch-sensitive surface,navigation is accomplished with a pointing device in other embodiments.Indeed, in some embodiments, there is no requirement that the computingdevice have a touch-sensitive surface at all. In those embodiments, theuser may provide input, equivalent to a gesture, using a pointingdevice.

In some embodiments, keyboard selections are used to navigatepresentation elements of symbolic expressions in addition to or insteadof touch gestures and/or pointing device actions. For example, the usermay perform a touch gesture or pointing device action to set focus on aparticular presentation element of a symbolic expression and then usethe left, right, up, or down arrow keys to navigate to a nextpresentation element relative to current presentation element.

The exemplary user interfaces depicted in FIGS. 4A-4P include a “currentfocus,” which is not visually depicted. Many of these exemplary userinterfaces also include a visually displayed “accessibility cursor”. Theaccessibility cursor indicates where the current focus is. The currentfocus is associated with a specific portion of a symbolic expressionwhich, if selected, would cause the device to execute an actionassociated with that specific portion. For example, in UI 400A (FIG. 4A)both the displayed accessibility cursor 402 and the current focus are onor at a portion of a symbolic expression 401, so that if that portionwere selected, the device running the user interface would aurallypresent information associated with the portion 401.

Although in some embodiments, as exemplified by UI 400A, the portionwhich has the current focus is the entire symbolic expression (e.g., theentire quadratic formula), the portion which has the current focus maybe less than the entire symbolic expression in other embodiments. Forexample, in UI 400B (FIG. 4B) both the accessibility cursor 402 and thecurrent focus are on or at the fraction portion 401 of the quadraticequation, so that if the fraction portion were selected, the devicerunning the user interface would aurally present information associatedwith the fraction portion 401 of the quadratic equation.

Although in some embodiments an accessibility cursor is displayedvisually in the user interface, the accessibility cursor is notdisplayed visually in other embodiments. Indeed, in some embodiments,there is not a requirement that a user interface is visually displayedat all. In those embodiments, audible and/or tactile feedback (e.g.,Braille, key-boards) which may be communicatively attached to ordisposed in the device, is provided to a user so that the user mayinterface elements according to methods disclosed herein. In theexemplary user interfaces in FIGS. 4A-4P, however, the current focus isat or on the same portion where the accessibility cursor 402 ispositioned (when the accessibility cursor is displayed).

In addition to symbolic expressions, the exemplary user interfacesdepicted in FIGS. 4A-P may display other content (e.g., text, images,and/or video) which is visually depicted in some of the user interfaces.For example, in UI 400A, other content is represented as boxes 403 inwhich a symbolic expression is presented in line with the other content.Although in some embodiments a symbolic expression is presented in linewith other content, a symbolic expression is surrounded by, behind, orin front of other content in other embodiments. Although in someembodiments other content is displayed in the user interface, othercontent is not displayed in other embodiments. For example, a userinterface may display just a symbolic expression or just a portion of asymbolic expression without displaying other content. While some of theexemplary user interfaces depicted in FIGS. 4A-P display other content,those user interfaces are not required to display other content, unlessthe context clearly indicates otherwise. Similarly, the exemplary userinterfaces depicted in FIGS. 4A-P that do not display other content arenot required to omit display of other content, unless the contextclearly indicates otherwise.

In some embodiments, some or all of the exemplary gestures contained inTable 1 may be utilized in accessibility interfaces as set forth herein.Each of the exemplary gestures in Table 1 is categorized into one of twodifferent types of navigation. In some embodiments, the exemplarygestures are performed to navigate portions of a symbolic expressionusing location-independent user gestures. In some embodiments, theexemplary gestures are performed to navigate portions usinglocation-dependent user gestures.

TABLE 1 Exemplary accessibility gestures Location-independent navigationDetected gesture Response to detected gesture Single-finger tap on thetouch-sensitive Speak information associated with the portion surface.of the symbolic expression where the current focus is set. Single-fingerflick right on the touch sensitive Move the current focus to the nextnavigable surface. portion of the symbolic expression. Single-fingerflick left on the touch-sensitive Move the current focus backward to thesurface. previous navigable portion of the symbolic expression.Double-finger flick right on the touch-sensitive Move the current focusto the next navigable surface. portion of the symbolic expression andincrease the selection granularity to the next highest level.Double-finger flick left on the touch-sensitive Move the current focusto the previous surface. navigable portion of the symbolic expressionand decrease the selection granularity to the next lowest level.Three-finger flick right on the touch-sensitive Move the current focusto the next navigable surface. portion of the symbolic expression,increase the selection granularity to the next highest level, andincrease the zoom level of the display screen. Three-finger flick lefton the touch-sensitive Move the current focus to the previous surface.navigable portion of the symbolic expression, decrease the selectiongranularity to the next lowest level, and decrease the zoom level of thedisplay screen. Two-finger single tap on the touch-sensitive Toggleoutput of current accessibility surface. information, e.g., togglesbetween pausing and playing audible accessibility information.Location-dependent navigation Detected gesture Response to Detectedgesture Single-finger tap on, near, at, or within a Set the currentfocus to the portion of the selectable area of the touch-sensitivesurface. symbolic expression corresponding to the selectable area andspeak information associated with the portion of the symbolic expressionwhere the current focus is now set. Single-finger trace acrossselectable areas of Set the current focus to the portion of the thetouch-sensitive surface. symbolic expression corresponding theselectable area being traced across and speak information associatedwith the portion where the current focus is now set.

The accessibility gestures in Table 1 are merely exemplary. In someembodiments, the response to opposite gestures (e.g., a downward gestureversus the corresponding upward gesture, or a leftward gesture versusthe corresponding rightward gesture) may be reversed from those shown inTable 1. For example, a single-finger flick left gesture may move thecurrent focus to the next navigable portion and a single-finger flickright gesture may move the current focus to the previous navigableportion. In some embodiments, the responses to opposite gestures areuser configurable, e.g., via a settings or options menu.

In some embodiments, the accessibility user interface allows a user toassociate a command with a predefined gesture. In some embodiments, agesture may be combined with a modified key (e.g., a control, options,command, or shift key) and mapped to any command of the user's choice.For example, a user may choose to map a two-finger rotation gesture withthe control-key activated to a command that adjusts the speaking rate ofthe accessibility information.

Location-Independent Gestures for Sequential Navigation

UI 400A-UI 400G (FIGS. 4A-4G) depict an example of advancing through asequence of portions of a symbolic expression displayed on one userinterface screen.

UI 400A (FIG. 4A) illustrates an exemplary user interface displaying anavigable symbolic expression. In this example, the symbolic expressionis the quadratic formula, but could just as easily by anothermathematical symbolic expression or a chemical or scientific symbolicexpression. The accessibility cursor 402 is at the quadratic formulaportion 401 as a whole, so the current focus is on or at the quadraticformula portion 401 as a whole.

Drill-Down Operation

User gesture 404, e.g., a finger swipe or flick gesture, has an initialcontact 404-1 on the touch screen 112 that moves 404-2 towards the rightedge of the touch screen 112. Note that in UI 400A, exemplary usergesture 404 is independent of contacting quadratic equation portion 401.

UI 400C (FIG. 4C) illustrates the exemplary user interface followinguser gesture 404. The accessibility cursor has moved from the quadraticequation as a whole to the identifier portion 401 in response to usergesture 404. This indicates that the current focus is now at theidentifier portion 401 instead of the quadratic equation as a whole.Audible information associated with the identifier portion 401, such as“x” and/or “left-hand side”, is also output in response to user gesture404. In some embodiments, the audible information also includes adrill-down sound to indicate that the current focus has moved from aportion of the symbolic expression to a sub-portion of that portion inresponse to user gesture 404. In this example, the identifier portion401 is a sub-portion of the entire quadratic equation.

While the user gesture 404 in the exemplary embodiment of UI 400A (FIG.4A) is a single-finger touch gesture, in other embodiments the usergesture 404 is a multi-finger touch gesture such as a two-finger flickright gesture or three-finger flick-right gesture.

Although in UI 400C the current focus moved from the quadratic equationas a whole to the identifier portion 401 in response to the user gesture404, the current focus could have just as easily moved from thequadratic equation as a whole to another sub-portion of the quadraticequation in response to the user gesture 404. For example, the currentfocus could have moved to the sub-portion consisting of the x identifierand the = (equals) operator, or moved to the sub-portion consisting ofthe entire fraction portion of the quadratic equation, or any othersub-portion of the quadratic equation.

Tree-Based Representation of Symbolic Expressions

In some embodiments, navigable portions of a symbolic expression arerepresented in a computer memory of the device as a tree data structure.The tree data structure has a root node and one or more other nodes thatare children of one other node. Each node of the tree represents aportion of the symbolic expression, with the root node representing alargest portion of the symbolic expression (e.g., the entire symbolicexpression).

While in some embodiments the root node represents the entire symbolicexpression (e.g., the entire quadratic equation), in other embodimentsthe root node represents a sub-symbolic expression of the entiresymbolic expression (e.g., the right-hand side of the quadraticequation). Each node in the tree corresponds to a portion of thesymbolic expression that can be the subject of the current focus. A nodecan have a parent node (in which case that node is also a child node ofthe parent node), one or more child nodes (in which case that node isalso a parent node of the one or more child nodes), and/or one or moresibling nodes (in which case that node and its sibling nodes all havethe same parent node). A child node represents a sub-portion of theportion of the symbolic expression represented by its parent node. Aparent node represents a super-portion of the portions of the symbolicexpression represents respectively by its children.

In some embodiments, the tree representation facilitates advancingthrough navigable portions of a symbolic expression in a sequencecorresponding to either a depth-first or breadth-first traversal of thetree. For example, FIG. 5A illustrates an exemplary tree representation500A of the quadratic equation. Tree 500A has twenty-one nodes numberedfrom 501-1 to 501-21 in breadth-first traversal order. If the currentfocus is set to node 501-5, then a gesture to advance the current focuswould set the current focus at node 501-6. Audible informationassociated with node 501-6 such as “denominator” and/or “2 times a”, isalso output in response to the advance gesture. If the current focus isset at node 501-8, then a gesture to move the current focus backward byone navigable unit would set the current focus at node 501-7. Audibleinformation associated with node 501-7 such as “minus b”, is also outputin response to the move backward gesture.

In contrast, tree 500B of FIG. 5B has the same twenty-one nodes numberedfrom 502-1 to 502-21 in depth-first traversal order. In this case, ifthe current focus is set to node 502-5, then a gesture to advance thecurrent focus would set the current focus at node 502-6 of tree 500B.Audible information associated with node 502-6 of tree 500B such as“minus b”, is also output in response to the advance gesture. If thecurrent focus is set at node 502-9, then a gesture to move the currentfocus backward by one navigable unit would set the current focus at node502-8. Audible information associated with node 502-8 of tree 500B suchas “b”, is also output in response to the move backward gesture.

Although in some embodiments advancing or moving backward navigableportions of a symbolic expression corresponds to a depth-first orbreadth-first traversal of a tree representation of the symbolicexpression, advancing or moving backward navigable portions maycorrespond to other traversals of the tree representation. Indeed, agesture to advance or move backward can correspond to a traversal fromthe node where the current focus is set directly to any other node ofthe tree representation. For example, if the current focus is set atnode 501-4 of tree representation 500A (FIG. 5A), a gesture to advanceto the next navigable portion may correspond to a traversal directly tonode 501-9 or directly to any other node of the tree representation500A.

Although not a requirement, the portions of a symbolic expression thatare considered the navigable portions may be based on a markup languagedescription of the symbolic expression from which a tree representationof the symbolic expression may be generated. For example, in themathematics domain, a Mathematical Markup Language (MathML) or LaTexdescription may be used. In other domains, other markup languagedescriptions may be used.

In FIGS. 5A and 5B, each node of tree representations 500A and 500Bcorrespond to a portion of a symbolic expression. Although not depictedin tree representations 500A and 500B, in some embodiments, eachindividual token and layout element of a symbolic expression correspondsto a leaf node in a tree representation of the symbolic expression. Forexample, referring to tree representation 500A, individual elements ±and √{square root over ( )} do not correspond to a leaf node inrepresentation 500A or 500B. However, in other embodiments, theindividual elements ± and √{square root over ( )} each correspond to aleaf node that is a child of node 501-5 in tree representation 500A anda child of node 502-5 in tree representation 500B. What is considered tobe an individual element of a symbolic expression may vary according tothe requirements of the implementation at hand. For example, what isconsidered to be an individual element of a symbolic expression may bebased on a markup language description or other description of thesymbolic expression such as, for example, a MathML or LaTex descriptionof the symbolic expression.

Navigating Between Sub-Portions of an Equation

Returning to the navigation example, UI 400C (FIG. 4C) depicts usergesture 406 on the exemplary user interface, e.g., another single-fingeror multi-finger swipe or flick gesture. Gesture 406 has an initialcontact 406-1 on the touch screen 112 that moves 406-2 towards the rightedge of the touch screen 112. Note that exemplary user gesture 406remains independent of contacting identifier portion 401 or any otherportion of the symbolic expression.

UI 400D (FIG. 4D) depicts that, in response to user gesture 406, thecurrent focus, and thus, the accessibility cursor 402 has moved to theequals operator portion 401 of the quadratic equation. Audibleinformation associated with the equals operator portion 401, such as“operator” and/or “equals”, is also output in response to user gesture406.

UI 400D (FIG. 4D) also depicts user gesture 408 on the exemplary userinterface, e.g., another single-finger or multi-finger left-to-rightfinger swipe or flick gesture. Gesture 408 has initial contact point408-1 on the touch screen that moves 408-2 towards the right edge of thetouch screen 112. Note that exemplary user gesture 408 remainsindependent of contacting equals portion 401 or any other portion of thesymbolic expression.

UI 400E (FIG. 4E) depicts that, in response to user gesture 408, thecurrent focus, and thus, the accessibility cursor 402 has moved to thefraction portion 401 of the quadratic equation. Audible informationassociated with the fraction portion 401, such as “right-hand side”,“fraction” or “the fraction minus b plus or minus the square root of bsquared minus 4 times a times c over 2 times a”, is also output inresponse to user gesture 408.

UI 400E (FIG. 4E) also depicts user gesture 410 on the exemplary userinterface, e.g., another single-finger or multi-finger left-to-rightfinger swipe or flick gesture. Gesture 410 has initial contact point410-1 on the touch screen that moves 410-2 towards the right edge of thetouch screen 112. Note that exemplary user gesture 410 remainsindependent of contacting fraction portion 401 or any other portion ofthe symbolic expression.

UI 400F (FIG. 4F) depicts that, in response to user gesture 410, thecurrent focus, and thus, the accessibility cursor 402 has moved from thefraction portion 401 of the quadratic equation to the numerator portion401 of the fraction portion. Audible information associated with thenumerator portion 401, such as “numerator” and/or “minus b plus or minusthe square root of b squared minus 4 times a times c”, is also output inresponse to user gesture 410.

In some embodiments, the audible information also includes a drill-downsound to indicate that the current focus has moved from a portion of thesymbolic expression to a sub-portion of that portion in response to usergesture 410. In this example, the current focus has moved from thefraction portion of the quadratic equation to the numerator portion ofthe fraction. A user may continue to advance through the navigableportions of the symbolic expression by continuing to input single-fingeror multi-finger left-to-right finger swipe or flick gestures (notshown).

While in some embodiments as exemplified in UI 400C-400F (FIGS. 4C-4F),the same gesture is performed to navigate between portions of a symbolicexpression within a level and to drill-down from one level to another,one gesture is performed to navigate between portions of a symbolicexpression within a level and another different gesture is performed todrill-down from one level to another in other embodiments. In theseembodiments, an audible navigation indicator sound may be output whenthe gesture to navigate between portions of a symbolic expression withina level is performed when there are no more portions in the level in thedirection of the gesture to navigate to. In some embodiments, thelocation-independent gesture to navigate between portions of a symbolicexpression within a level is a single-finger flick or swipe gesture andthe location-independent gesture to drill-down from one level to anotheris a two-finger flick or swipe gesture. For example, referring to UI400E (FIG. 4E), assuming user gesture 410 is a single-finger flick orswipe gesture, in response to user gesture 410, an audible beep or otherhalting sound may be output to indicate to the user that there are nomore portions of the symbolic expression in the current level in thedirection of the gesture 410 to navigate to. In this case, in responseto the single-finger flick or swipe gesture 410, the current focusremains at portion 402. On the other hand, assuming user gesture 410 isa two-finger flick or swipe gesture, in response to that gesture 410,the current focus may move from the fraction portion 401 of thequadratic equation to the numerator portion 401 of the fraction portionas shown in UI 400F (FIG. 4F). Thus, one gesture may be performed tonavigate between sibling portions a symbolic expression and anotherdifferent gesture may be performed to drill-down from a parent portionto child portion of the symbolic expression.

Expand-Up Operations

UI 400F (FIG. 4F) also depicts user gesture 412 on the exemplary userinterface, e.g., a single-finger or multi-finger right-to-left fingerswipe or flick gesture. Gesture 412 has initial contact point 412-1 onthe touch screen that moves 412-2 towards the left edge of the touchscreen 112 which is substantially opposite of gestures 404, 406, and408. Note that exemplary user gesture 412 remains independent ofcontacting portion 401 or any other portion of the symbolic expression.

UI 400G (FIG. 4G) depicts that, in response to user gesture 412, thecurrent focus, and thus, the accessibility cursor 402 has moved back tothe fraction portion 401 of the quadratic equation. Audible informationassociated with the fraction portion 401, such as “right-hand side”,“fraction”, or “the fraction minus b plus or minus the square root of bsquared minus 4 times a times c over 2 times a”, is also output inresponse to user gesture 412.

In some embodiments, the audible information also includes an expand-upsound to indicate that the current focus has moved from a portion of thesymbolic expression to a super-portion of that portion in response touser gesture 412. In this example, the current focus has moved from thenumerator portion as depicted in UI 400F (FIG. 4F) to the fractionportion as depicted in UI 400G (FIG. 4G). The expand-up sound may be thesame as or different from the drill-down sound.

While in some embodiments as exemplified in UI 400E-400G (FIGS. 4F and4G), the same gesture is performed to navigate between portions of asymbolic expression within a level and to expand-up from one level toanother, one gesture is performed to navigate between portions of asymbolic expression within a level and another different gesture isperformed to expand-up from one level to another in other embodiments.

Accordingly, location-independent user gestures in one direction mayadvance the current focus through a given sequence or order of portionsof a symbolic expression; while location-independent user gestures in asubstantially opposite direction may cause the current focus to retreator backtrack through the given sequence or order of portions of thesymbolic expression. Advancing or backtracking through a sequence ofportions of a symbolic expression may include drilling-down intosub-portions and expanding-up into super-portions of the symbolicexpression. Different gestures may be used to advance or backtrackthrough portions in a group of portions and to drill-down or expand-upfrom one portion to another. For example, single-finger gestures may beused to advance or backtrack sibling portions of a symbolic expressionand multi-finger gestures may be used to drill-down or expand-up betweenparent and child portions.

Location-Dependent Gestures for Selective Navigation

UI 400H-UI 400I (FIGS. 4H-4I) depict an example of moving the currentfocus and the accessibility cursor using location-dependent gestures. Inthis example, the accessibility cursor 402 (and current focus) isinitially on or at some other content 403 (e.g., text). User gesture413, depicted in UI 400H, is a stationary contact on the touch screen112, e.g., a single-finger tap on the quadratic equation portion 401.

UI 400I (FIG. 4I) depicts that, in response to user gesture 413 (FIG.4H), the current focus, and thus, the accessibility cursor 402 moves tothe quadratic equation portion 401. Audible information associated withthe quadratic equation portion 401, such as “quadratic equation” and/or“x equals the fraction minus b plus or minus the square root of bsquared minus 4 times a times c over 2 times a”, is also output inresponse to user gesture 413. Thus, the accessibility user interface mayinclude both location-dependent finger gestures for navigation thatprovide accessibility information about a portion of a symbolicexpression at the location of the finger gesture (e.g., gesture 413), aswell as location-independent finger gestures for navigation that provideaccessibility information about a portion of a symbolic expressionindependent of the location of the finger gesture (e.g., gestures 404,406, 408, 410, and 412).

In some embodiments, the audible information associated with a portionof a symbolic expression that is output in response to a user gesturecontains grouping information to audibly indicate logical groupings ofportions to the user. For example, in response to user gesture 413 (FIG.4H), audible information associated with the quadratic equation portion401, such as “x equals start fraction 1 start numerator negative b plusor minus the square root of start group 1b squared minus 4 times a timesc end group 1 end numerator start fraction 1 denominator 2 times a endfraction 1 denominator end fraction 1”, may be output.

Location-Independent and Location-Dependent Gestures in Combination

UI 400I-UI 400N (FIGS. 4I-4N) depict an example of navigating portionsof a symbolic expression displayed on one user interface screen using acombination of location-independent and location-dependent gestures. Inthis example, location-independent user gestures are performed toadvance and backtrack through a sequence of navigable portions of asymbolic expression as described and depicted in UI 400A-UI 400G (FIGS.4A-4G). In addition, location-independent gestures are performed tochange the “selection granularity” of the symbolic expression. The“selection granularity” determines what portions of the symbolicexpression are selectable through performance of location-dependent usergestures.

The exemplary user interfaces depicted in FIGS. 4I-4N include one ormore selectable areas are associated with a current selectiongranularity. Each selectable area covers an area of the display screenand is associated with a specific portion of a symbolic expressionwhich, if selected, would cause the device to execute an actionassociated with that specific portion. The specific portion associatedwith a selectable area may be selected by the user by performing alocation-dependent gesture on, at, near, or within the selectable area.For example, in UI 400I (FIG. 4I) there is one selectable area 414associated with the entire quadratic equation portion 401, so that if alocation-dependent gesture is performed on, at, near, or within theselectable area 414, the device running the user interface would aurallypresent information associated with the entire quadratic equationportion 401.

Although in some embodiments, as exemplified in UI 400I, a selectablearea may coincide with the area of the accessibility cursor 402 (whetheror not displayed), a selectable area may not coincide with the area ofthe accessibility cursor 402 in other embodiments. Although in someembodiments a selectable area is indicated visually in the userinterface, a selectable area is not indicated visually in otherembodiments.

UI 400I (FIG. 4I) illustrates an exemplary user interface displaying anavigable symbolic expression. In this example, the symbolic expressionis the quadratic formula, but could just as easily by anothermathematical symbolic expression or a chemical or scientific symbolicexpression. The accessibility cursor 402 is at the quadratic formulaportion 401 as a whole, so the current focus is on or at the quadraticformula portion 401 as a whole.

Increasing Granularity of Selectable Sub-Parts

A location-independent user gesture 415, e.g., a single-finger ormulti-finger swipe or flick gesture, has an initial contact 415-1 on thetouch screen 112 that moves 415-2 towards the right edge of the touchscreen 112. Note that in UI 400I, exemplary user gesture 415 isindependent of contacting quadratic equation portion 401.

UI 400J (FIG. 4J) illustrates the exemplary user interface followinguser gesture 415. The accessibility cursor has moved from the quadraticequation as a whole to the identifier portion 401 in response to usergesture 415. This indicates that the current focus is now at theidentifier portion 401 instead of the quadratic equation as a whole.Audible information associated with the identifier portion 401, such as“x” and/or “left-hand side”, is also output in response to user gesture415. In some embodiments, the audible information also includes adrill-down sound to indicate that the current focus has moved from aportion of the symbolic expression to a sub-portion of that portion inresponse to user gesture 415. In this example, the identifier portion401 is a sub-portion of the entire quadratic equation.

Also in response to user gesture 415, the current selection granularityhas become finer (i.e., increased) relative to the previous selectiongranularity in effect prior to user gesture 415. As a result, inresponse to user gesture 415, there are now three selectable areas414-1, 414-2, and 414-3 corresponding to the left-hand side, the equalsoperator, and the right-hand side of the quadratic equationrespectively. In other words, in response to user gesture 415,selectable area 414-1 in effect prior to user gesture 415, has beenreplaced by selectable areas 414-1, 414-2, and 414-3.

While in some embodiments as exemplified in UI 400I-400J (FIGS. 4F and4G), the same gesture is performed to navigate between portions of asymbolic expression within a granularity level and to drill-down orexpand-up from one granularity level to another, one gesture isperformed to navigate between portions of a symbolic expression within agranularity level and another different gesture is performed todrill-down or expand-up from one granularity level to another in otherembodiments. For example, referring to UI 400I (FIG. 4I), in order toincrease the granularity level, the user gesture 415 may be required tobe a multi-finger swipe or flick gesture. In this case, if user gesture415 is a single-finger swipe or flick gesture, then audible navigationsound (e.g., a beep or bump sound) may be output to indicate that thereare no more portions to navigate to in the current granularity level inthe direction of the single-finger swipe or flick.

Characteristics of Selectable Areas

In some embodiments, a selectable area for an associated portion of asymbolic expression has the shape of a box (square or rectangle) withrounded or right-angle corners that surrounds the portion of thesymbolic expression associated with the selectable area as displayed onthe display screen but that does not overlap with other selectable areasfor other portions. Thus, the size and shape of the selectable area mayvary depending on the size of the portion, the current display screenzoom level, font size, screen orientation, and other display and/orcontent factors.

In some embodiments, a selectable area is expanded vertically and/orhorizontally to cover whitespace in the area surrounding the portion ofthe symbolic expression with which the selectable area is associated.For example, in UI 400J (FIG. 4J), the selectable areas 414-1 and 414-2have been expanded vertically to match the vertical height of selectablearea 414-3.

While in some embodiments selectable areas are abutting, selectableareas may have a border or boundary such that adjacent selectable areashave a whitespace gap between them.

In some embodiments, the selectable area for a portion of a symbolicexpression corresponds to a layout bounding box that encloses theportion in a display screen layout of the symbolic expression. Forexample, the layout bounding box may correspond to a markup languageelement (e.g., a HTML element) of a markup language document that islaid out on the display screen. In these embodiments, as the position,shape and size of the layout bounding box changes, the position, shape,and size of the selectable area is also changed correspondingly tomaintain coincidence between the layout bounding box and the selectablearea. For example, if the display screen is zoomed such that the size ofthe layout bounding box is changed, then the corresponding selectablearea is also changed correspondingly.

While in some embodiments a selectable area has the shape of a box withrounded or right-angle corners, a selectable area has the shape of anoval, a circle, or other polygon in other embodiments.

While in example user interface of FIG. 4J (and other example userinterfaces of the figures), adjacent selectable areas are depicted asnot abutting each other, in other embodiments adjacent selectable areasare abutting each other.

Increasing Granularity of Selectable Sub-Parts Continued

A location-dependent gesture 416, depicted in UI 400J, is a stationarycontact on the touch screen 112, e.g., a single-finger tap on, at, near,or within selectable area 414-3.

UI 400K (FIG. 4K) depicts that, in response to location-dependent usergesture 416 (FIG. 4J), the current focus, and thus, the accessibilitycursor 402 moves to the fraction portion 401 of the quadratic equation.Audible information associated with the fraction portion 401, such as“fraction” or “the fraction minus b plus or minus the square root of bsquared minus 4 times a times c over 2 times a”, is also output inresponse to user gesture 416. The current selection granularity has notchanged in response to user gesture 416. Accordingly, there are stillthe three selectable areas 414-1, 414-2, and 414-3.

UI 400K (FIG. 4K) also depicts location-independent user gesture 417 onthe exemplary user interface, e.g., another swipe or flick gesture.Gesture 417 has an initial contact 417-1 on the touch screen 112 thatmoves 417-2 towards the right edge of the touch screen 112. Note thatexemplary user gesture 417 is independent of contacting any portion ofthe symbolic expression.

UI 400L (FIG. 4L) depicts that, in response to user gesture 417, thecurrent focus, and thus, the accessibility cursor 402 has moved to thenumerator portion 401 of the quadratic equation. Audible informationassociated with the numerator portion 401, such as “numerator” and/or“minus b plus or minus the square root of b squared minus 4 times atimes c”, is also output in response to user gesture 417. In someembodiments, the audible information also includes a drill-down sound toindicate that the current focus has moved from a portion of the symbolicexpression to a sub-portion of that portion in response to user gesture417. In this example, the numerator portion 401 is a sub-portion of thefraction portion.

Also in response to user gesture 417, the current selection granularityhas become finer (i.e., increased) relative to the previous selectiongranularity in effect prior to user gesture 417 (UI 400K). As a result,in response to user gesture 417, there are now four selectable areas414-1, 414-2, 414-3, and 414-4 corresponding to the left-hand side, theequals operator, the numerator, and the denominator of the quadraticequation respectively. Note, in this example, selectable areas 414-1 and414-2 corresponding to the left-hand-side and the equals operatorrespectively were not changed in response to user gesture 417. However,selectable area 413-1 in in effect prior to user gesture 417 (UI 400K)was, in response to user gesture 417, replaced by selectable areas 414-3and 414-4 as shown in UI 400L. A location-dependent gesture 418,depicted in UI 400L, is a stationary contact on the touch screen 112,e.g., a single-finger tap on, at, near, or within selectable area 414-4.

In some embodiments, in response to user gesture 417, the currentselection granularity is increased but there are only two selectableareas 414-3 and 414-4 as shown in UI 400L (FIG. 4L) instead of fourselectable areas and the selectable areas 414-1 and 414-2 as shown in UI400L (FIG. 4L) are not selectable at the new selection granularity. Inthis case, the user may be perform a location-independent gesture (e.g.,a two-finger swipe or flick left gesture) to decrease the selectiongranularity and restore the selectable areas 414-1, 414-2, and 414-3 asshown in UI 400K (FIG. 4K).

UI 400M (FIG. 4M) depicts that, in response to location-dependent usergesture 418 (FIG. 4L), the current focus, and thus, the accessibilitycursor 402 moves to the denominator portion 401 of the quadraticequation. Audible information associated with the denominator portion401, such as “denominator” and/or “2 times a”, is also output inresponse to user gesture 418. The current selection granularity has notchanged in response to user gesture 418. Accordingly, there are stillthe four selectable areas 414-1, 414-2, 414-3, and 414-4. A user maycontinue to advance through the navigable portions of the symbolicexpression by continuing to input left-to-right finger swipe or flickgestures (not shown). A user may continue to select portions of thesymbolic expression by continuing to input single-tap gestures toselectable areas corresponding to the portions.

Expand-Up Operations

UI 400M (FIG. 4M) also depicts location-independent user gesture 419 onthe exemplary user interface, e.g., a right-to-left swipe or flickgesture. Gesture 419 has an initial contact 419-1 on the touch screen112 that moves 419-2 towards the left edge of the touch screen 112. Notethat exemplary user gesture 419 is independent of contacting any portionof the symbolic expression.

UI 400N (FIG. 4N) depicts that, in response to user gesture 419, thecurrent focus, and thus, the accessibility cursor 402 has moved back tothe fraction portion 401 of the quadratic equation. Audible informationassociated with the fraction portion 401, such as “right-hand side”,“fraction”, or “the fraction minus b plus or minus the square root of bsquared minus 4 times a times c over 2 times a”, is also output inresponse to user gesture 419. In some embodiments, the audibleinformation also includes an expand-up sound to indicate that thecurrent focus has moved from a portion of the symbolic expression to asuper-portion of that portion in response to user gesture 419. In thisexample, the current focus has moved from the denominator portion asdepicted in UI 400M (FIG. 4M) to the fraction portion as depicted in UI400N (FIG. 4N). The expand-up sound may be the same as or different fromthe drill-down sound.

Also in response to user gesture 419, the current selection granularityhas become coarser (i.e., decreased) relative to the previous selectiongranularity in effect prior to user gesture 419 (UI 400M). As a result,in response to user gesture 419, there are three selectable areas 414-1,414-2, and 414-3 corresponding to the left-hand side, the equalsoperator, and the right-hand side of the quadratic equationrespectively. Note, in this example, selectable areas 414-1 and 414-2corresponding to the left-hand-side and the equals operator respectivelywere not changed in response to user gesture 419. However, selectableareas 414-3 and 414-4 in effect prior to user gesture 419 (UI 400M)were, in response to user gesture 419, replaced by selectable area 414-3as shown in UI 400N.

Accordingly, location-independent user gestures in one direction, aswell as causing the current focus to advance through a given sequence ororder of portions of a symbolic expression, may cause the selectiongranularity to increase; while location-independent user gestures in asubstantially opposite direction, as well as causing the current focusto retreat or backtrack through the given sequence or order of portionsof the symbolic expression, may cause the selection granularity todecrease. Advancing or backtracking through a sequence of portions of asymbolic expression may include drilling-down into sub-portions andexpanding-up into super-portions of the symbolic expression.

In some embodiments, instead of a single-finger flick right gesture toadvance to the next portion of a symbolic expression and increase theselection granularity, a two-finger flick right gesture is performed. Insome embodiments, instead of a single-finger flick left gesture tobacktrack to the previous portion of a symbolic expression and decreasethe selection granularity, a two-finger flick left gesture is performed.

In some embodiments, a three-finger flick right gesture, as well asadvancing to the next portion and increasing the selection granularity,also increases the zoom level of the display screen. In someembodiments, a three-finger flick left gesture, as well as backtrackingto the previous portion and decreasing the selection granularity, alsodecreases the zoom level of the display screen.

Selection Granularity and Tree Representation of a Symbolic Expression

In some embodiments, a tree representation of a symbolic expressionstored in computer memory is involved in determining which portions ofthe symbolic expression correspond to selectable areas at the currentselection granularity. Each level of the tree corresponds to a differentselection granularity. The topmost level of the tree consisting of justthe root node corresponds to the lowest (i.e., most coarse) selectiongranularity. The bottommost level that includes the most nested nodes ofthe tree corresponds to the highest (i.e., finest) selectiongranularity. The current selection granularity is at the level of thetree where the current focus is set. Every node in the tree at the levelwhere the current focus is set corresponds to portions of the symbolicexpression that correspond to selectable areas at the current selectiongranularity. In addition, all leaf nodes of the tree (i.e., nodes thathave no children) at levels in the tree higher that the level where thecurrent focus is set also correspond to portions of the symbolicexpression that correspond to selectable areas at the current selectiongranularity.

For example, tree representation 503 of FIG. 5C is of the symbolicequation f(x)=(x+1)⁴. Initially, the current focus is at the portion ofthe symbolic represented by node 503-1 of the tree. Accordingly, thecurrent selection granularity is set at the first level (Level 1) of thetree 503. The portion of the symbolic expression corresponding to node503-1 corresponds to a selectable area at the current selectiongranularity as indicated with the bolded box outline in FIG. 5C.

Tree 503 in FIG. 5D shows that the current focus has moved to theportion of the symbolic expression corresponding to node 503-2.Accordingly, the current selection granularity is set at the secondlevel (Level 2) of the tree 503. The portions of the symbolic expressioncorresponding to nodes 503-2, 503-4, and 503-5 correspond to selectableareas at the current selection granularity as indicated with bolded boxoutlines in FIG. 5D.

Tree 503 in FIG. 5E shows that the current focus has moved to theportion of the symbolic expression corresponding to node 503-5.Accordingly, the current selection granularity is set at the secondlevel (Level 3) of the tree 503. The portions of the symbolic expressioncorresponding to nodes 503-2, 503-4, 503-5, and 503-6 correspond toselectable areas at the current selection granularity as indicated withbolded box outlines in FIG. 5E.

Tree 503 in FIG. 5F shows that the current focus has moved to theportion of the symbolic expression corresponding to node 503-8.Accordingly, the current selection granularity is set at the third level(Level 4) of the tree 503. The portions of the symbolic expressioncorresponding to nodes 503-2, 503-4, 503-7, 503-8, 503-9, and 503-6correspond to selectable areas at the current selection granularity asindicated with bolded box outlines in FIG. 5F.

Changing Screen Reader Behavior

In some embodiments, a common gesture may be used to change settings forhow accessible navigation of a symbolic expression is performed (e.g.,in response to right or left single-flick or swipe gestures, Table 1).For example, a two-finger rotation or twisting gesture on thetouch-sensitive surface may be used to “turn” a virtual dial, rotor, orknob to choose between options (referred to hereinafter as “root” or“rotor control” and gestures of this type may be referred to herein as“rotor gestures”). This “rotor” control may act as an invisible dial forchanging the response of the device to the detection of certainnavigation gestures in the accessibility user interface.

The effect of the rotor control can vary. As one example, the rotor maychange the response of the device to the detection of a single-flick orswipe right/left gestures from one response option to another. Threepossible response options are: (1) moving the current focus to thenext/previous navigable portion of the symbolic expression; (2) movingthe current focus to the next/previous navigable portion of the symbolicexpression and increasing/decreasing the selection granularity; and (3)moving the current focus to the next/previous navigable portion of thesymbolic expression, increasing/decreasing the selection granularity,and increasing/decreasing the current zoom level of the display screen.As another example, the rotor may move the current focus to the next orprevious navigable portion of the symbolic expression. In this case, aclockwise rotational pattern may advance the current focus to the nextnavigable portion while a counter-clockwise rotational pattern may movethe current focus backwards to the previous navigable portion (or viceversa).

In some embodiments, a rotor-based gesture includes a first fingercontact and a second finger contact that are separated by a distance,where the distance between the first finger contact and the secondfinger contact includes a center point, and the first finger contact andthe second finger contact rotate on the touch screen display around thecenter point. In essence, as discussed above, this rotation gesturesimulates turning or twisting a knob in one direction or the other. Asthe gesture progresses, the simulated knob turning changes the gestureresponse or moves to the current focus.

In some embodiments, a rotor-based gesture includes placing a firstfinger contact on a touch-sensitive surface, concurrently placing asecond finger contact on the touch-sensitive surface, and rotating thesecond finger contact about the first finger contact while the firstfinger contact remains stationary. In this embodiment, the first fingercontact acts as a center point for the rotation by the second fingercontact.

A rotor-based gesture is just one example of a common gesture that maybe used to change settings for how accessible navigation of a symbolicexpression is performed. Other common gestures may be used to accomplishthe same. As just one example, a two-finger double-tap gesture may beused to accomplish the same as a rotor-based gesture in aclockwise-rotational pattern and a single-finger double-tap gesture maybe used to accomplish the same as a rotor-based gesture in acounter-clockwise rotational pattern, or vice-versa. As another example,a spread gesture may be used to accomplish the same as a rotor-basedgesture in a clockwise-rotational pattern and a pinch gesture may beused to accomplish the same as a rotor-based gesture in acounter-clockwise rotational pattern, or vice-versa.

In some embodiments, a common gesture is used to select how audibleinformation corresponding to the portion of the symbolic expression withthe current focus is output. For example, a common gesture may be usedto select between a literal explanation of the portion of the symbolicexpression and a more semantic explanation of the portion. For example,a literal explanation of the portion x² may be audible information suchas “x raised to the power of two”, while a more semantic explanation ofthe portion x² may be audible information such as “x squared”.

In some embodiments, a common gesture is used to select the audibleoutput style of the audible information of the portion of the symbolicexpression with the current focus. For example, a common gesture may beused to select between a mathematical style and chemical style.

In some embodiments in which the computing device is operatively coupledto a braille output device, a common gesture is used to select the typeof braille output to the braille output device. For example, a commongesture may be used to select between Nemeth braille math output andBritish math braille output.

While in some embodiments a touch gesture is used to change settings forhow accessible navigation of a symbolic expression is performed, akeystroke or set of keystrokes is used to change the settings in otherembodiments.

Tracing Gestures

UI 400O and UI 400P (FIG. 4O and FIG. 4P) illustrate that, in someembodiments, the accessibility user interface facilitates the use of afinger in a tracing motion on the touch-sensitive surface to navigatethrough and activate portions of a symbolic expression presented in theinterface. In UI 400O of FIG. 4O, user selection gesture 421 begins witha first point of contact 421-1 on at, near, or within selectable area420-2. In response, the current focus is placed at the portion of thesymbolic expression corresponding to selectable area 420-2 (i.e., x) andaudible information about the corresponding portion is emitted, e.g.,“x” and/or “left-hand side”. As user selection gesture 421 begins tomove 421-2 across the touch-sensitive surface, the current focus movesto other portions of the symbolic expression. In the example of UI 400O,user selection gesture 421 leaves 421-2 selectable area 420-2 and moves421-2 across selectable areas 420-3, 420-4, and 420-5 corresponding tothe equals operator portion, the right-hand side portion, and thefraction bar portion of the symbolic expression respectively, so thecurrent focus is identified as being the equals operator portion,followed by the right-hand side portion, followed by the fraction barportion as the user selection gesture 421 moves 421-2 across thetouch-sensitive surface. Audible information about the equals operatorportion is emitted, e.g., “equals”, as or when the user selectiongesture 421 enters 421-2 or moves 421-2 on, at, near, or withinselectable area 420-3. Audible information about the right-hand sideportion is emitted, e.g., “right-hand side”, “fraction”, or “thefraction minus b plus or minus the square root of b squared minus 4times a times c over 2 times a”, as or when the user selection gesture421 enters 421-2 or moves 421-2 on, at, near, or within selectable area420-4. Audible information about the fraction bar portion is emitted,e.g., “fraction bar” or a sound indicating a faction bar, as or when theuser selection gesture 421 enters 421-2 or moves 421-2 on, at, near, orwithin selectable area 420-5.

In some embodiments, an audible transition sound is emitted as or whenthe user selection gesture leaves one selectable area (e.g., movesacross a boundary of the selectable area from a point on, at, near, orwithin the selectable area to a point that is not on, at, near, orwithin the selectable area) and enters another selectable area (e.g.,moves across a boundary of the selectable area from a point that is noton, at, near, or within the selectable area to a point that is on, at,near, or within the selectable area). For example, referring to UI 400O,as or when the user selection gesture 421 leaves 421-2 selectable area420-1 and enters 421-2 selectable area 420-3, an audible transitionsound such as a beep, whirl, click, or other sound is emitted.

Further, the transition sound may be different depending on whether thetransition is from one selectable area to another selectable area nestedwith that selectable area or from one selectable area to anotherselectable area that contains that selectable area. For example, in UI400P of FIG. 4P, user selection gesture 422 begins with a first point ofcontact 422-1 on at, near, or within selectable area 420-1. In response,the current focus is placed at the portion of the symbolic expressioncorresponding to selectable area 420-1 (i.e., the entire quadraticequation) and Audible information about the corresponding portion isemitted, e.g., “symbolic expression”, “equation”, “quadratic equation”,and/or “x equals the fraction minus b plus or minus the square root of bsquared minus 4 times a times c over 2 times a”. As user selectiongesture 422 begins to move 422-2 across the touch-sensitive surface, thecurrent focus moves to other portions of the symbolic expression.

In the example of UI 400P, user selection gesture 422 leaves selectablearea 420-1 and enters 422-2 selectable area 420-4 corresponding to theright-hand side portion, so the current focus is identified as being theright-hand side portion and a transition sound followed by audibleinformation associated with the right-hand side portion are emitted. Thetransition sound may be distinctive such that it indicates to the user(e.g., through pitch, pitch variance, number of distinct sounds, typesof sound, etc.) that a nested selectable area is being entered by theuser selection gesture 422 (in this example, selectable area 420-4 isnested within selectable area 420-1). The transition sound may alsoindicate the level of nesting of the selectable area being enteredrelative to the outermost selectable area (in this example, theselectable area 420-4 being entered is one level nested relative tooutermost selectable area 420-1). Audible information about theright-hand side portion is emitted, e.g., “right-hand side”, “fraction”,or “the fraction minus b plus or minus the square root of b squaredminus 4 times a times c over 2 times a”, as or when the user selectiongesture 422 enters 422-2 or moves 422-2 on, at, near, or withinselectable area 420-4.

Continuing the example of UI 400P, user selection gesture 422 leavesselectable area 420-4 and enters 422-2 selectable area 420-7corresponding to the numerator portion, so the current focus isidentified as being the numerator portion and a transition soundfollowed by audible information associated with the numerator portionare emitted. The transition sound may be distinctive such that itindicates to the user (e.g., through pitch, pitch variance, number ofdistinct sounds, types of sound, etc.) that a nested selectable area isbeing entered by the user selection gesture 422 (in this example,selectable area 420-7 is nested within selectable area 420-4). Thetransition sound may also indicate the level of nesting of theselectable area being entered relative to the outermost selectable area(in this example, the selectable area 420-7 being entered is two levelsnested relative to outermost selectable area 420-1). Audible informationabout the numerator portion is emitted, e.g., “numerator” or “minus bplus or minus the square root of b squared minus 4 times a times c”, asor when the user selection gesture 422 enters 422-2 or moves 422-2 on,at, near, or within selectable area 420-7.

Continuing the example of UI 400P, user selection gesture 422 leavesselectable area 420-7 and enters 422-2 selectable area 420-11corresponding to the square root portion, so the current focus isidentified as being the square root portion and a transition soundfollowed by audible information associated with the square root portionare emitted. The transition sound may be distinctive such that itindicates to the user (e.g., through pitch, pitch variance, number ofdistinct sounds, types of sound, etc.) that a nested selectable area isbeing entered by the user selection gesture 422 (in this example,selectable area 420-11 is nested within selectable area 420-7). In thiscase, the transition sound may also indicate the level of nesting of theselectable area being entered relative to the outermost selectable area(in this example, the selectable area 420-11 being entered is threelevels nested relative to outermost selectable area 420-1). Audibleinformation about the square root portion is emitted, e.g., “squareroot” or “the square root of b squared minus 4 times a times c”, as orwhen the user selection gesture 422 enters 422-2 or moves 422-2 on, at,near, or within selectable area 420-11.

Continuing the example of UI 400P, user selection gesture 422 leavesselectable area 420-11 and enters 422-2 selectable area 420-12corresponding to operand of the square root, so the current focus isidentified as being the operand of the square root and a transitionsound followed by audible information associated with the operand of thesquare root portion are emitted. The transition sound may be distinctivesuch that it indicates to the user (e.g., through pitch, pitch variance,number of distinct sounds, types of sound, etc.) that a nestedselectable area is being entered by the user selection gesture 422 (inthis example, selectable area 420-12 is nested within selectable area420-11). In this case, the transition sound may also indicate the levelof nesting of the selectable area being entered relative to theoutermost selectable area (in this example, the selectable area 420-12being entered is four levels nested relative to outermost selectablearea 420-1). Audible information about the operand of the square root isemitted, e.g., “b squared minus 4 times a times c”, as or when the userselection gesture 422 enters 422-2 or moves 422-2 on, at, near, orwithin selectable area 420-12. Although not shown, the individual tokens(e.g., b, 4, a, c) or groupings of individual tokens (e.g., b² or 4ac)of the square root operand could each also be within their ownrespective selectable areas nested with selectable area 420-12.

Continuing the example of UI 400P, user selection gesture 422 leaves422-2 selectable area 420-12 and re-enters 422-2 selectable area 420-11corresponding to the square root portion, so the current focus isidentified as being the square root portion and a transition soundfollowed by audible information associated with the square root portionare emitted. The transition sound may be distinctive such that itindicates to the user (e.g., through pitch, pitch variance, number ofdistinct sounds, types of sound, etc.) that a containing selectable areais being entered by the user selection gesture 422 (in this example,selectable area 420-11 contains selectable area 420-12). In this case,the transition sound may also indicate the level of nesting of thecontaining selectable area being entered relative to the outermostselectable area (in this example, the containing selectable area 420-11being entered is three levels nested relative to outermost selectablearea 420-1). Audible information about the square root portion isemitted, e.g., “square root” or “the square root of b squared minus 4times a times c”, as or when the user selection gesture 422 enters 422-2or moves 422-2 on, at, near, or within selectable area 420-12.

Continuing the example of UI 400P, user selection gesture 422 leaves422-2 selectable area 420-11 and re-enters 422-2 selectable area 420-17corresponding to the numerator portion, so the current focus isidentified as being the numerator portion and a transition soundfollowed by audible information associated with the numerator portionare emitted. The transition sound may be distinctive such that itindicates to the user (e.g., through pitch, pitch variance, number ofdistinct sounds, types of sound, etc.) that a containing selectable areais being entered by the user selection gesture 422 (in this example,selectable area 420-7 contains selectable area 420-11). In this case,the transition sound may also indicate the level of nesting of thecontaining selectable area being entered relative to the outermostselectable area (in this example, the containing selectable area 420-7being entered is two levels nested relative to outermost selectable area420-1). Audible information about the numerator portion is emitted,e.g., “numerator” or “minus b plus or minus the square root of b squaredminus 4 times a times c”, as or when the user selection gesture 422enters 422-2 or moves 422-2 on, at, near, or within selectable area420-7.

Continuing the example of UI 400P, user selection gesture 422 leaves422-2 selectable area 420-7 and re-enters 422-2 selectable area 420-4corresponding to the right-hand side portion, so the current focus isidentified as being the right-hand side portion and a transition soundfollowed by audible information associated with the right-hand sideportion are emitted. The transition sound may be distinctive such thatit indicates to the user (e.g., through pitch, pitch variance, number ofdistinct sounds, types of sound, etc.) that a containing selectable areais being entered by the user selection gesture 422 (in this example,selectable area 420-4 contains selectable area 420-7). In this case, thetransition sound may also indicate the level of nesting of thecontaining selectable area being entered relative to the outermostselectable area (in this example, the containing selectable area 420-4being entered is one level nested relative to outermost selectable area420-1). Accessibility information about the right-hand side portion isemitted, e.g., “right-hand side”, “fraction”, or “the fraction minus bplus or minus the square root of b squared minus 4 times a times c over2 times a”, as or when the user selection gesture 422 enters 422-2 ormoves 422-2 on, at, near, or within selectable area 420-4.

Continuing the example of UI 400P, user selection gesture 422 leaves422-2 selectable area 420-4 and enters 422-2 selectable area 420-5corresponding to the fraction bar, so the current focus is identified asbeing the fraction bar and a transition sound followed by audibleinformation associated with the fraction bar portion are emitted. Thetransition sound may be distinctive such that it indicates to the user(e.g., through pitch, pitch variance, number of distinct sounds, typesof sound, etc.) that a nested selectable area is being entered by theuser selection gesture 422 (in this example, selectable area 420-5 isnested within selectable area 420-4). In this case, the transition soundmay also indicate the level of nesting of the selectable area beingentered relative to the outermost selectable area (in this example, theselectable area 420-5 being entered is two levels nested relative tooutermost selectable area 420-1). Accessibility information about thefraction bar is emitted, e.g., “fraction bar” or a sound that representsa fraction bar, as or when the user selection gesture 422 enters 422-2or moves 422-2 on, at, near, or within selectable area 420-5.

Continuing the example of UI 400P, user selection gesture 422 leaves422-2 selectable area 420-5 and re-enters 422-2 selectable area 420-4corresponding to the right-hand side portion, so the current focus isidentified as being the right-hand side portion and a transition soundfollowed by audible information associated with the right-hand sideportion are emitted. The transition sound may be distinctive such thatit indicates to the user (e.g., through pitch, pitch variance, number ofdistinct sounds, types of sound, etc.) that a containing selectable areais being entered by the user selection gesture 422 (in this example,selectable area 420-4 contains selectable area 420-5). In this case, thetransition sound may also indicate the level of nesting of thecontaining selectable area being entered relative to the outermostselectable area (in this example, the containing selectable area 420-4being entered is one level nested relative to outermost selectable area420-1). Accessibility information about the right-hand side portion isemitted, e.g., “right-hand side”, “fraction”, or “the fraction minus bplus or minus the square root of b squared minus 4 times a times c over2 times a”, as or when the user selection gesture 422 enters 422-2 ormoves 422-2 on, at, near, or within selectable area 420-4.

Continuing the example of UI 400P, user selection gesture 422 leaves422-2 selectable area 420-4 and enters 422-2 selectable area 420-6corresponding to the denominator portion, so the current focus isidentified as being the denominator and a transition sound followed byaudible information associated with the denominator portion are emitted.The transition sound may be distinctive such that it indicates to theuser (e.g., through pitch, pitch variance, number of distinct sounds,types of sound, etc.) that a nested selectable area is being entered bythe user selection gesture 422 (in this example, selectable area 420-6is nested within selectable area 420-4). In this case, the transitionsound may also indicate the level of nesting of the selectable areabeing entered relative to the outermost selectable area (in thisexample, the selectable area 420-6 being entered is two levels nestedrelative to outermost selectable area 420-1). Accessibility informationabout the denominator is emitted, e.g., “denominator” and/or “2 timesa”, as or when the user selection gesture 422 enters 422-2 or moves422-2 on, at, near, or within selectable area 420-6.

Although in some embodiments audible information associated with aportion of a symbolic expression corresponding to a selectable area isemitted both when the user selection gesture enters the selectable areafrom a containing selectable area and when the user selection gestureenters the selectable area from a nested selectable area, audibleinformation is emitted only when the user selection gesture enters theselectable area from a containing selectable area in other embodiments.For example, referring to UI 400P, audible information associated withthe numerator portion corresponding to selectable area 420-7 may beemitted when the user selection gesture 422 leaves 422-1 selectable area420-4 and enters selectable area 420-7 but not when the user selectiongesture 422 leaves selectable area 420-7 and enters selectable area420-4. However, in these embodiments, a transition sound is stillemitted both when the user selection gesture enters the selectable areafrom a containing selectable area and when the user selection gestureenters the selectable area from a nested selectable area.

While in the exemplary user interfaces 400O and 400P (and in otherexemplary user interfaces of the figures), a selectable area nestedwithin another selectable area is depicted as not sharing any portion ofits border with the containing selectable area (e.g., selectable area420-12 is depicted in UI 400O as not sharing any portion of its borderwith the containing selectable area 420-11), a selectable area nestedwithin another selectable area does share at least one border with itscontaining selectable area in other embodiments.

While in embodiments described above audible information is output as atracing gesture enters and leaves a selectable area, in otherembodiments audible information is output as a tracing gesture isperformed within a selectable area. In particular, audible informationmay be output as a tracing gesture performed within a selectable areatraces the display area of a symbol of the symbolic expression displayedwithin the selectable area. For example, referring to UI 400O, as userselection gesture 421 traces 421-2 the fraction symbol displayed inselectable area 420-5 an audible tick, click, or other sound may beoutput while the user selection gesture 422 is tracing 422-2 the symbol.The audible information may be output continuously while the userselection gesture 422 is tracing the 422-2 the symbol. In this way, atracing gesture may be performed that traces a symbol of a symbolicexpression to acquire a spatial layout of that symbolic through audibleinformation that is continuously and concurrently output as the tracingof the symbol is being performed.

In some embodiments as exemplified in UI 400O and 400P (FIGS. 40 and4P), each selectable area 414 corresponds to a portion of a symbolicexpression. Although not depicted in UI 400O and 400P, in someembodiments, each individual token and layout element of a symbolicexpression corresponds to a separate selectable area. For example, thenumber 2 and the symbol a in selectable area 414-8 may each reside intheir own separate selectable areas that are sub-selectable areas ofselectable area 414-8.

In some embodiments, audible information is not output for selectableareas below the current selection granularity level. For example, whentracing gesture 421 of UI 400O is performed; audible information foronly selectable areas 414-1, 414-3, and 414-4 may be output. Similarly,when tracing gesture 422 of UI 400P is performe; audible information foronly selectable areas 414-4, 414-5, and 414-6 may be output.

Mapping Selectable Areas Corresponding to Portions of a SymbolicExpression to a Touch-Sensitive Surface

FIGS. 6A-6F illustrate accessibility user interfaces that map respectiveselectable areas corresponding to portions of a symbolic expression to atouch-sensitive surface in accordance with some embodiments. Theselectable area mappings are part of an efficient accessibility methodfor navigation through complex three-dimensional symbolic expressions. Aprinted form of a symbolic expression can require a reader to read andunderstand the symbolic expression in a horizontal dimension (e.g., leftto right), a vertical dimension (e.g., up and down), and layereddimension (e.g., nested sub-expression) in order to understand theexpression as a whole. For example, to read and understand the symbolicexpression

$x = \frac{{- b} \pm \sqrt{b^{2} - {4a\; c}}}{2a}$

as the quadratic equation, the reader must read and understand theexpression in a horizontal dimension to know it is an equation, in avertical dimension to understanding the right-hand side is a fraction,and in a layered dimension to under that b²−4ac is a square rootoperand.

As illustrated herein, a selectable area corresponding to a portion of asymbolic expression is a discrete region of a user interface on adisplay. Selectable areas contain corresponding portions of the symbolicexpression. Selectable areas may be organized in a hierarchical fashion(e.g., nested within each other) corresponding to a hierarchicalarrangement of the corresponding portions of the symbolic expression.

In some embodiments, the accessibility user interfaces with selectableareas operate on a computing device with a display and a touch-sensitivesurface (e.g., a track pad), such as a desktop computer or a laptopcomputer. In some embodiments, the accessibility user interfaces withselectable areas operate on a computing device with a touch-screendisplay. In some embodiments, the touch screen display may be usedwithout one or more additional touch-sensitive surfaces, or one or moreadditional displays. In some embodiments, the accessibility userinterfaces with selectable areas are used on portable computing devices(e.g., 100, FIG. 1A) or other computing devices (e.g., 300, FIG. 3).

In FIGS. 6A-6F, gestures are depicted on a touch sensitive surface 601(e.g., a track pad) that is separate from the display. That theexemplary gestures are illustrated in these figures on a separatetouch-sensitive surface 601 does not require that the gestures beperformed on a track pad (or other separate touch-sensitive surface) topractice the techniques disclosed herein. For example, in someembodiments, the gestures may be performed on a touch screen instead.For these embodiments, a selectable area of current focus may be mappedto the entire touch screen surface while the touch screen continues todisplay multiple selectable areas, including the selectable area ofcurrent focus.

Table 2 describes exemplary accessibility gestures for mapped selectablearea navigation. These gestures are typically finger gestures. In someembodiments, the accessibility gestures for selectable areas may alsoinclude other inputs (e.g., keyboard inputs such as holding down amodifier key, or mouse inputs) in conjunction with or in place of thefinger gestures.

TABLE 2 Exemplary accessibility gestures for mapped selectable areanavigation Detected Gesture Response to Detected Gesture Two-fingerswipe Select a next lower selectable area associated right on the touch-with where the current focus is set. sensitive surface. Two-finger swipeleft Select a next higher selectable area associated on thetouch-sensitive with where the current focus is set. surface.Single-finger moving/ Move the current focus in the current dragging onthe touch- selectable area in accordance with the mapping sensitivesurface. of the current selectable area on the touch- sensitive surfaceand the current position of the single-finger on the touch-sensitivesurface.

In addition, navigational gestures discussed above with respect to Table1 (e.g., location-independent gestures, location-dependent gestures,flick navigation, trace navigation, etc.) area also applicable to mappedselectable area navigation.

The accessibility gestures in Table 2 are merely exemplary. In someembodiments, a two-finger spread (depinch) gesture, instead of atwo-finger swipe right gesture, is used to select a next lower (nested)selectable area associated with where the current focus is set. In someembodiments, a two-finger pinch gesture, instead of a two-finger swipeleft gesture, is used to select a next higher (containing) selectablearea associated with where the current focus is set. In someembodiments, the response to opposite gestures (e.g., a downward gestureversus the corresponding upward gesture, or a leftward gesture versusthe corresponding rightward gesture) may be reversed from those shown inTable 2. For example, a two-finger flick left gesture may be used toselect a next lower (nested) selectable area and a two-finger flickright gesture may be used to select a next higher (containing)selectable area. In some embodiments, the responses to opposite gesturesare user configurable, e.g., via a settings or options menu.

FIG. 6A depicts a mapping of a selectable area corresponding to aportion of a symbolic expression to touch-sensitive surface 601. In someembodiments, touch sensitive surface 601 is touch pad 360 on device 300(FIG. 3).

In the example of FIG. 6A, selectable area 614-1 and the selectableareas 614-2 through 614-12 contained therein are each proportionallymapped to touch-sensitive surface 601. The proportional mappingvertically stretches and horizontally compresses each of the selectableareas 614-1 through 614-12 as necessary to fit the area of thetouch-sensitive surface 601. Vertical or horizontal stretching makes iteasier for a visually impaired user to touch a location on thetouch-sensitive surface 601 that corresponds to a given selectable area.

UI 600B-UI 600F (FIGS. 6B-6F) depict exemplary user interfaces fornavigating a hierarchical (nested) arrangement of selectable areascorresponding to hierarchical (nested) portions of a symbolicexpression.

Referring to UI 600B (FIG. 6B), current selectable area indicator 615-1(e.g., a bolded or otherwise visually highlighted border) is used toidentify that the entire symbolic expression is presently selected. Insome embodiments, as depicted here, current selectable area indicatormay border the selectable area. In some embodiments, current selectablearea indicator borders the portion of the symbolic expression to whichthe selectable area corresponds. Other techniques for visuallyhighlighting the current selectable area are possible. For example, thecurrent selectable area may be filled in with a transparent highlightcolor. As another example, the portion of the symbolic expressioncorresponding to the current selectable area may be highlighted (e.g.,the color of the characters and symbols changes from black to ahighlight color).

In this example, selectable area 614-1 is proportionally mapped 616 totouch-sensitive surface 601 because the current selectable area is theoverall symbolic expression.

User interface selectable area selection event 617, in this example, atwo-finger swipe gesture, has initial points of contact 617-1 and 617-2,following by movement (617-3 and 617-4) of these initial points ofcontact. Note that user interface selectable area selection event 617 islocation independent. Event 617 may occur at an arbitrary angle andlocation on the touch-sensitive surface 601. In contrast, gestures fornavigating and activating within a given selectable area (e.g., a singlefinger tracing gesture) are location-dependent gestures on the mappedtouch-sensitive surface.

UI 600C (FIG. 6C) illustrates that in response to detecting userinterface selectable area selection event 617, current selectable areaindicator 615-2 has moved to a nested selectable area of the previouscurrently selected selectable area 614-1 to reflect that selectable area614-4 is now the currently selected selectable area mapped to thetouch-sensitive surface 601. In response to user interface selectablearea selection gesture 617, selectable area 614-4 outlined by currentselectable area indicator 615-2 is proportionally mapped 618 totouch-sensitive surface 601. The device may also output audibleinformation associated with the portion of the symbolic expressionassociated with the currently selected selectable area 614-4 such asspeaking “right-hand side”, “fraction”, or “the fraction minus b plus orminus the square root of b squared minus 4 times a times c over 2 timesa”. The device may also output a drill-down sound or a transition soundin response to user interface selectable area selection gesture 617.

Note, as exemplified by the current example, nested selectable areas maybe skipped in response to a user interface selectable area selectionevent such as event 617. For example, selectable areas 614-2 and 614-3nested within containing selectable area 614-1 were skipped in responseto event 617. This skipping may be performed because the skippedselectable areas do not have any or have only a few nested selectableareas.

UI 600C also includes an exemplary user interface navigation gesture619, in this example, a single-finger trace on the touch-sensitivesurface 601, that has initial point of contact 619-1, following bymovement 619-2 of the initial point of contact.

UI 600D (FIG. 6D) illustrates movement of the current focus 602 withinthe current selectable area 614-4 in accordance with the user interfacenavigation gesture 619 (FIG. 6C). Here, the current selectable area614-4 corresponds to the right-hand side of the quadratic equation. InUI 600D, the position of the current focus 602 corresponds to thelocation of the moving single finger contact 619-3 on the mappedtouch-sensitive surface 601. The device may also output audibleinformation associated with the current focus 602 in response todetecting the user interface navigation gesture 629, such as speaking“square root operand” or “b squared minus 4 times a times c”.

Thus, the current focus may move independently of the current selectedselectable area mapped to the touch-sensitive surface. Further, theaccessibility user interface may include both location-independentfinger gestures for navigating between nested selectable areas, andlocation-dependent finger gestures for navigation within a currentlyselected selectable area, where the current selected selectable area ismapped to the touch-sensitive surface. Both types of gestures may alsocause audible information to be output.

Referring now to UI 600E (FIG. 6E), user interface selectable areaselection event 620, in this example, a two-finger swipe gesture, hasinitial points of contact 620-1 and 620-2, following by movement (620-3and 620-4) of these initial points of contact. Note that user interfaceselectable area selection event 620 is location independent. Event 620may occur at an arbitrary angle and location on the touch-sensitivesurface 601. In contrast, gestures for navigating and activating withina given selectable area (e.g., a single finger tracing gesture) arelocation-dependent gestures on the mapped touch-sensitive surface.

UI 600F (FIG. 6F) illustrates that in response to detecting userinterface selectable area selection event 620, current selectable areaindicator 615-3 has moved to the containing selectable area 614-1 of theprevious currently selected selectable area 614-4 to reflect thatselectable area 614-1 is now the currently selected selectable area. Inresponse to user interface selectable area selection gesture 620,selectable area 614-1 outlined by current selectable area indicator615-3 is proportionally mapped 621 to touch-sensitive surface 601. Thedevice may also output audible information associated with the portionof the symbolic expression associated with the currently selectedselectable area 614-1 such as speaking “expression”, “equation”,“quadratic equation”, or “the equation x equals the fraction minus bplus or minus the square root of b squared minus 4 times a times c over2 times a”. The device may also output an expand-up sound or atransition sound in response to user interface selectable area selectiongesture 620.

Note, as exemplified by the current example, sibling selectable areasmay be skipped in response to a user interface selectable area selectionevent such as event 620. For example, selectable areas 614-2 and 614-3siblings of selectable area 614-4 were skipped in response to event 620.This skipping may be performed because the skipped selectable areas donot have any or have only a few nested selectable areas. Although notillustrated, containing selectable areas may also be skipped in responseto a user interface selectable area selection event. For example, inresponse to a user interface selectable area selection event, a parentselectable area may be skipped moving directly to a grandparent,great-grandparent, or great-great grandparent, etc. selectable area.

A Process for Presenting a Symbolic Expression

FIG. 7 is a flow diagram illustrating an accessibility method fornavigation among portions of a symbolic expression in accordance withsome embodiments. The method 700 is performed at an electronic devicesuch as a computing device (e.g., 300, FIG. 3) with a display inconjunction with one or more programs (e.g., accessibility module 129,operating system 126 and/or one or more applications 134). Someoperations in method 700 may be combined and/or the order of someoperations may be changed.

The method 700 provides an efficient way to navigate through portions ofa symbolic expression and provide audible information about theportions. The method reduces the cognitive burden on a user withimpaired vision when using a computing device with a display. Forbattery-operated devices, enabling a user to navigate faster and moreefficiently conserves power and increases the time between batterycharges.

The device displays (702) a symbolic expression on the display (e.g., UI400A). The display of the symbolic expression includes display of afirst portion of the symbolic expression and display of a second portionof the symbolic expression. The first portion and the second portion maybe mutually exclusive portions of the symbolic expression (i.e., the twoportions do not overlap in content). For example, the denominatorportion of the quadratic equation 2a is mutually exclusive of thenumerator portion −b±√{square root over (b²−4ac)}. The first portion maybe a part of the second portion. For example, the denominator portion ofthe quadratic equation 2a is part of the fraction portion

$\frac{{- b} \pm \sqrt{b^{2} - {4a\; c}}}{2a}.$

The second portion may be part of the first portion. For example, thedenominator portion of the quadratic equation 2a is part of the fractionportion

$\frac{{- b} \pm \sqrt{b^{2} - {4a\; c}}}{2a}.$

A current focus is on the first portion of the symbolic expression.Accordingly, when the current focus is on the first portion of thesymbolic expression, audible information related to or describing thefirst portion may be output (704) upon detecting appropriate user input.The user input may be touch sensitive surface-based (e.g., fingergesture), pointing device-based (e.g., mouse), and/or keyboard-based(e.g., striking a key or holding down a key). The audible informationmay be a spoken description of the first portion, a navigation contextsound (e.g., a beep, whirl, whistle, chime, etc.), and/or spokennavigation signpost (e.g., the current selection granularity level).

User input is detected (706) that selects a second portion of thesymbolic expression for aural presentation. The user input may be touchsensitive surface-based (e.g., finger gesture), pointing device-based(e.g., mouse), and/or keyboard-based (e.g., striking a key or holdingdown a key). For touch sensitive surface-based input, the user inputgesture may be location-dependent (e.g., a single-finger trace),location-independent (e.g., a single-finger swipe gesture, Table 1),and/or a combination of location-independent and location-dependentgestures.

In response to the user input selecting the second portion, the currentfocus transitions (708) from the first portion to the second portion.Information associated with the second portion is determined (708) andaurally presented (710). The audible information may be a spokendescription of the second portion, a navigation context sound (e.g., abeep, whirl, whistle, chime, etc.), and/or spoken navigation signpost(e.g., the current selection granularity level).

In some embodiments, at least a portion of the information associatedwith the second portion that is aurally presented is determined based ona tree representation of the symbolic expression. For example, adescription of the second portion may be associated with a node of thetree representation corresponding to the second portion. The descriptionmay be in a text data format that can be input to a text-to-speechtransducer or may be ready-to-play digital audio content (e.g., a WAV orAAC file).

In some embodiments, multiple descriptions of the second portion areassociated with the second portion and one is selected and aurallypresented. Which of the multiple descriptions is selected may vary basedon context and/or user settings. For example, the multiple descriptionsmay correspond to different verbosity levels and one of the multipledescriptions may be selected based on a currently selected verbositylevel that is selected, for example, using a rotor gesture or other userinput. As another example, the multiple descriptions may correspond todifferent learning/domain sophistication levels and one of the multipledescriptions may be selected based on a currently selected learninglevel selected, for example, using a rotor gesture or other user input.As yet another example, the multiple descriptions may correspond todifferent users or different types of users and one of the multipledescriptions may be selected based on the identity of the current user.For example, if the identity of the current user indicates that the useris a student (e.g., based on configuration data stored in memory of thedevice), then the description selected may be a student-orientateddescription. Alternatively, if the identity of the current userindicates that the user is a professional scientist, then thedescription selected may be a professional-orientated description.

Extensions and Alternatives

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

1. A computer-implemented method, comprising: at a computing device witha display: displaying a first portion of a symbolic expression within afirst area of the display; while in a first state in which the firstarea is selected for aural presentation, aurally presenting firstinformation related to the first portion of the symbolic expression;while in the first state, detecting particular user input; in responseto detecting the particular user input: transitioning from the firststate to a second state in which a second area of the display isselected for aural presentation; determining second informationassociated with a second portion of the symbolic expression that isdisplayed within the second area; in response to determining the secondinformation, aurally presenting the second information.
 2. The method ofclaim 1, wherein the computing device has a touch-sensitive surface; andwherein detecting the particular user input includes detecting a userinterface navigation gesture on the touch-sensitive surface.
 3. Themethod of claim 2, wherein the display is a touch-screen display andwhere the touch-sensitive surface is on the display.
 4. The method ofclaim 2, wherein the user interface navigation gesture is a multi-fingergesture.
 5. The method of claim 2, wherein the user interface navigationgesture is independent of contacting a location on the touch-sensitivesurface that corresponds to the second area.
 6. The method of claim 2,wherein the user interface navigation gesture does not contact alocation on the touch-sensitive surface that corresponds to the secondarea.
 7. The method of claim 2, wherein the user interface navigationgesture is dependent on contacting a location on the touch-sensitivesurface that corresponds to the second area.
 8. The method of claim 2,further comprising: while in the first state, mapping the first area ofthe display to the touch-sensitive surface; and in response to detectingthe particular user input: ceasing to map the first area of the displayto the touch-sensitive surface; and proportionally mapping the secondarea of the display to be substantially coextensive with thetouch-sensitive surface.
 9. The method of claim 1, wherein the symbolicexpression is a mathematical, scientific, or chemical expression,equation, or formula.
 10. The method of claim 1, wherein detecting theparticular user input includes detecting a pointing device action. 11.The method of claim 5, wherein the pointing device is a mouse.
 12. Themethod of claim 1, wherein detecting the particular user input includesdetecting a key strike on a keyboard.
 13. The method of claim 1, whereinthe second area resides within the first area and wherein the secondportion of the symbolic expression is a part of the first portion. 14.The method of claim 1, wherein the first information includes one ormore words, aurally presented as spoken text, describing the firstportion of the symbolic expression.
 15. The method of claim 1, whereinthe second information includes one or more words, aurally presented asspoken text, describing the second portion of the symbolic expression.16. The method of claim 1, further comprising: in response todetermining the second information, outputting a transition sound toindicate that the area selected for aural presentation has transitionedfrom the first area to the second area.
 17. The method of claim 1,wherein the first area resides within the second area and wherein thefirst portion of the symbolic expression is a part of the secondportion.
 18. A computing device, comprising: a touch-sensitive surface;a display; one or more processors; memory; and one or more programsstored in the memory and which, when executed by the one or moreprocessors, cause the computing device to: display a first portion of asymbolic expression within a first area of the display; while in a firststate in which the first area is selected for aural presentation,aurally present first information related to the first portion of thesymbolic expression; while in the first state, detect particular userinput; in response to detecting the particular user input: transitionfrom the first state to a second state in which a second area of thedisplay is selected for aural presentation; determine second informationassociated with a second portion of the symbolic expression that isdisplayed within the second area; in response to determining the secondinformation, aurally present the second information.
 19. One or morenon-transitory computer-readable media storing one or more programswhich, when executed by a computing device with a display, cause thecomputing device to perform a method comprising: displaying a firstportion of a symbolic expression within a first area of the display;while in a first state in which the first area is selected for auralpresentation, aurally presenting first information related to the firstportion of the symbolic expression; while in the first state, detectingparticular user input; in response to detecting the particular userinput: transitioning from the first state to a second state in which asecond area of the display is selected for aural presentation;determining second information associated with a second portion of thesymbolic expression that is displayed within the second area; inresponse to determining the second information, aurally presenting thesecond information.
 20. A graphical user interface on a computing devicewith a display, the graphical user interface comprising: a symbolicexpression displayed on the display; a plurality of selectable areas ofthe display; wherein: the symbolic expression has a plurality ofportions, each selectable area of the plurality of selectable areascorresponds to a portion of the plurality of portions; while in a firststate in which a first selectable area of the plurality of selectableareas is selected for aural presentation, first information associatedwith the portion of the symbolic expression corresponding to the firstselectable area is aurally presented; while in the first state, userinput selecting a second selectable area of the plurality of selectableareas is detected; and in response to detecting the user input, secondinformation associated with the portion of the symbolic expressioncorresponding to the second selectable area is aurally presented.
 21. Acomputer-implemented method, comprising: at a computing device: while ina first state in which a first portion of a symbolic expression isselected for presentation, presenting first information related to thefirst portion of the symbolic expression; while in the first state,detecting particular user input; in response to detecting the particularuser input: transitioning from the first state to a second state inwhich a second portion of the symbolic expression is selected forpresentation; determining second information associated with a secondportion of the symbolic expression; in response to determining thesecond information, presenting the second information.
 22. The method ofclaim 1, wherein the computing device has or is operatively coupled toan electro-mechanical braille display; and wherein presenting the secondinformation includes sending the information to the electro-mechanicalbraille display.